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Introduction to the Special Issue on the Education Doctorate 


James W. Guthrie 
Peabody College of Vanderbilt University 


David D. Marsh 


University of Southern California 


This Special Issue of the Peabody Journal of Education: Issues of Leadership, Policy & 
Organizations is devoted to what some have labeled “‘education’s dirty little secret,” the doctor 
of education degree, the Ed.D. 

We refer here to the regrettably weak nature of many advanced professional degree programs 
in educational leadership and administration and related fields. Literally hundreds of colleges and 
universities offer the doctor of education degree (Ed.D.), and annually issue tens of thousands 
of such degrees. Only a very few of these programs are rigorous intellectually or professionally 
useful when it comes to improving American education. 

The problem is that the Ed.D. degree, though often fulfilling some awardees’ needs for 
higher social status, does so without reciprocally requiring the high levels of knowledge and 
professional competency the general public expects of advanced professional degree recipients, 
such as in medicine, law, and engineering. 

Moreover, many Ed.D. programs operate outside the confines of conventional professional 
certification and, in effect, are diploma mills. Such institutions routinely exhibit low admissions 
standards, if they have any standards at all. Their faculty members seldom possess high levels of 
training and relevant expertise. They operate in a weak regulatory arena and offer enrollees the 
prospect of higher social and professional status in exchange for tuition. The implicit bargain is 
that the degree-issuing institution will require little of an Ed.D. candidate by way of intellectual 
and professional challenge if the recipient will pay the price of tuition. By the way, when it 
comes to tuition, federally subsidized, low-interest student loan programs are readily available 
with quite reasonable repayment terms. 

These weak, sometimes fraudulent, operations badly impugn the reputation of legitimate 
Ed.D.-offering institutions, ones that voluntarily adhere to high professional and intellectual 
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standards. Even at the best of our universities, Ed.D. programs are often hindered by tenure-track 
faculty who want the Ed.D. to be similar to a Ph.D. program and who are hesitant to involve 
field-relevant clinical faculty and other stakeholders to share in the program design and operation. 
Leaders of every Ed.D. program must also wrestle with problems of relevance to very challenging 
leadership issues in K-12 education. 

In the abstract, the Ed.D., invented approximately 100 years ago and first offered at the 
University of California, Berkeley, is intended to be the professional analog of the doctorate in 
medicine (M.D.), dentistry (D.O.D.), Divinity (D.D.), law (J.D.), etc. It is a professional degree, 
the awarding of which is intended to connote advanced research knowledge and unusually high 
levels of applied and craft competency, as well as ethical training, in the practice of the profession 
at hand. 

A reader needs to tread lightly when judging individual holders of the Ed.D. Selective programs 
maintain high standards, and particular graduates often exhibit high levels of professionalism and 
achieve high goals 

Reputable individual programs and individuals are not the target of our criticism. Our reform 
target is the larger field that is unable, principally because of pressures upon small institutions to 
survive economically, to engage in self-discipline and to impose upon its component parts higher 
expectations for professional performance. 

The problems with Ed.D. programs have not newly come to light, nor are we the first to lament 
the dismal nature of the Ed.D. Clifford and Guthrie wrote about the problem in their book Ed 
School in the 1970s. Levine described the troubled landscape in his commission report in 2005 
and institutional associations such as the American Association of Universities Education School 
Deans and the Carnegie Commission on Teaching have engaged in Ed.D. reform efforts for a 
decade. 

Regardless of such well-intentioned efforts, little has changed. As explained in the last article 
in this volume, it is not likely that change will occur until the problems of small colleges’ high 
dependence upon tuition is solved, in conjunction with rigorous professional accreditation and 
federal support for high-level education leadership development programs. 

This special Peabody Journal issue arises from the editors’ desires to illustrate to the larger 
higher education community the nature of a select few exemplary Ed.D. programs. The three 
programs featured here, and described in detail, are known for having announced admission 
standards, appropriately qualified faculty, and curricula offerings and program requirements 
linked to real-world challenges faced by practicing education leaders. Levine’s report and the 
Carnegie Commission have also featured these three programs as exemplary. 

The programs at the University of Southern California, St. Louis University, and Peabody 
College at Vanderbilt University are described in the following articles. These articles describe 
the overall programs at each institution and pay particular attention to the curriculum and to 
capstone projects (that replace research dissertations). These descriptive pieces appear after an 
introductory chapter that explains the challenges facing those desirous of upgrading the Ed.D. and "’ 
before a concluding chapter that offers a ray of hope for future correction of the field’s weaknesses. 
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The Case for a Modern Doctor of Education Degree 
(Ed.D.): Multipurpose Education Doctorates 
No Longer Appropriate 


James W. Guthrie 
Peabody College of Vanderbilt University 


Educational research and professional practice, though assuredly capable of providing reciprocal 
benefits one to the other, have, nevertheless, evolved into decidedly different activities. Increasingly, 
each has a separate and defensible technical basis. Doctoral programs in education should honor and 
reflect, not conflate, the differences. The Ed.D. is now deserving of prominent billing as the degree 
of choice for education practitioners. 


INTRODUCTION 


In some institutions, the education doctoral degree is a Ph.D. In others it is an Ed.D. Some 
institutions offer both. In a past era, a single-track, dual-purpose, research- and practitioner- 
integrated doctoral program was possible, even sensible. 

Old-style doctoral programs in prestigious institutions such as the University of Chicago, 
Teachers College, Harvard, and Stanford University routinely graduated individuals who with 
grace and competence could manage a state education department, school or school district, 
philanthropic foundation, high-level government or international program, or teach university 
courses. Illustrious individuals such as H. Thomas James, Roald Campbell, Harold (Doc) Howe, 
Frank Keppel, Roderick MacPhee, James A. Kelly, Joseph M. Cronin, Donna Shalala, Stephen P. 
Heyneman, and Larry Cuban were personifications of such multifaceted and multipurpose talent. 
They moved with ease from high levels of professional scholarship and instruction to program 
leadership, from classroom to boardroom, and back again. 

Less so today. Modern education research increasingly is characterized by a rigorous method- 
ological and philosophical paradigm entirely different than was true even in the late 20th century. 
Experimentation and large data set analyses, random and fixed effect modeling, are now the 
expected research mode. Measurement techniques such as those regularly used by epidemiolo- 
gists, psychologists, and economists, regression and discontinuity regression analysis, propensity 
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analysis, and hierarchical linear modeling are increasingly threshold quantitative skills for re- 
search methodological competency. These are skill sets and understandings that take time to 
impart, require immersion in analyses and research to perfect, and are not learned by lecture and 
from textbooks alone. 

That is only half the story. Now that the production of students’ academic performance 
(outputs), no longer the institution’s consumption of public resources (inputs), is the measure of K— 
12 school success, being an educational administrator is becoming a sophisticated professional and 
technical challenge, not simply a bureaucratically oriented or politically concentrated endeavor. 
This is becoming equally true of higher education management. 

Hence, education school doctoral degree programs that still attempt to meet all markets 
and cram both professional and research preparation into the same curriculum shortchange 
institutional obligations to enrollees and stigmatize education as a field. Failure to correct the 
shortcoming of one degree ensures continuance of weaknesses in both. 


ILLUSTRATING THE PROBLEM 


An analogy helps. Can physicians become researchers after attending medical school? Yes, but 
only in a limited sense. They can undertake selected analyses of patient treatments, sometimes 
even arranging for experimental treatment trials. However, they cannot delve into fundamental 
relationships between biochemical and physiological interactions, genetic research, or anatomical 
and mechanical processes without having far more intense disciplinary preparation than M.D. 
programs conventionally are able to provide. 

Education administration and educational research have, at last, evolved in the same research 
and methodological manner as medicine and engineering. An educational practitioner, having 
succeeded in a good Ed.D. program or a dual-purpose Ph.D. program, may well have acquired 
elementary research understandings. Such will enable him or her to test varying practices, record 
results carefully, and add to a literature of evidence-based professional practice. 

However, what she or he cannot easily do, not from personal deficiency but by virtue of limited 
academic preparation and examined experience, is design the detailed randomized field trials, 
statistically sophisticated data set analyses, and multiple cohort panel studies now characterizing 
acceptable education leadership and policy research. Such trainees cannot even meet advanced 
modern standards for case studies or survey analyses. 

Nor is a conventional education school—prepared practitioner qualified to undertake the sophis- 
ticated discipline-based studies now flowing from psychologists, political scientists, sociologists, 
or economists interested in fundamental properties of human learning, organizational dynamics, 
leadership, or matters of policy and productivity. 

A conventional educational administration doctoral program, of 3 or 4 years in duration, cannot 
transmit sufficient knowledge to prepare an individual both as an able practitioner and an able 
researcher. To the extent to which such a program operates on the tired time of the day, its evening 
and weekend classes catering to full-time employed practitioners, the challenge to high standards 
is even more daunting. 

A comparison with health or engineering helps again. An M.D., dentist, pharmacist, veteri- 
narian, or engineer pursuing a research career now typically seeks a Ph.D. in a related subject or 
cognate specialization. An M.D. would not likely be able, without intense research specialization, 
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to pass a Ph.D. qualifying examination in biochemistry, genetics, or physiology. Of course, he 
or she would also be unlikely to be attending graduate school part time while holding down a 
full-time job. Similarly, however, a Ph.D. candidate in, say, biochemistry or physiology, would 
be almost completely unprepared to sit for a physician’s various clinical and professional board 
examinations. 

The roles of researcher and practitioner are sufficiently different that reciprocal certification 
and role reversal would now be unthinkable, not to mention illegal. 

Here is a simple litmus test for the quality and propriety of a modern-era single-purpose 
research preparation program or a modern-era practitioner preparation program. An enrollee in 
either of such endeavors should not be sufficiently qualified by way of schooling or experience 
to pass the comprehensive examination or undertake the capstone experience or dissertation that 
the alternative program requires. 

If the comprehensive examinations or capstone requirements for research and practice are 
the same, program purposes, research preparation, and practitioner professional training have all 
been woefully compromised. 


EDUCATION NEEDS A MODERN Ed.D. DEGREE BECAUSE A 
DUAL-PURPOSE SINGLE-TRACK PROGRAM CANNOT NOW STAND 


Since when did being a successful practitioner involve such sophisticated skill sets? How did this 
happen anyway? Conversely, when did the underlying paradigm for education research undergo 
a transformation to an endeavor ever closer to science and how did that happen? 


“A Nation at Risk:” When Education in America Began to Matter 


One can trace an evolutionary line of contemporary education change from the 1983 release of 
“A Nation at Risk,’ through the 1989 Charlottesville Summit and its unprecedented generation of 
a set of National Educational Goals, to the 2001 enactment of No Child Left Behind. If one needs 
a triggering mechanism or causal explanation, concentrate on fundamental economic changes 
occurring throughout the world. 

From the time the United States was a colony until relatively recently, the nation has depended 
upon the capacities of a slender few educated elite to operate its economy, manage its government, 
direct its military, and maintain its universities. Most citizens were not well educated in a formal 
sense and, for narrow economic purposes, did not need to be. Johnny could drop out of school, 
get a job, and still earn a sufficient amount on an assembly line to marry Mary and achieve and 
maintain the material American dream. 

However, the late 20th-century coincidence of rapid travel and microchip electronics spelled 
the end of routinely available, well-paying, low-skilled jobs. New technologies fueled a new 
global economy wherein capital no longer recognizes national boundaries and where low-tech 
assembly line jobs, when they exist at all, are to be found offshore, seldom in America. 

Under such sophisticated conditions, a company’s or a person’s greatest fear is that of being 
transformed into a commodity, open to purchase in an open competitive market or at auction 
rather than individually selected for quality or skill. 
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Under the new economic regime, education matters, not just for the elite but for everyone. The 
path to pursue a better life and a chance for the American dream now winds its way through high 
school and into college. 

Most modern-day parents, particularly middle-class parents, recognize this changed condition, 
and they have expressed a political preference for better schools. Policy officials have responded, 
and No Child Left Behind (NCLB), much to some educators’ lament, is the new education reality. 
(Regardless of NCLB’s deficiencies, to wish that it go away entirely is tantamount to standing on 
the shore and ordering the waves to roll backwards.) 

NCLB symbolizes a dramatic turning point in American education. This legislation, however 
awkward in its initial implementation, marks the departure from judging schools and colleges by 
inputs alone. Now, it is what schools and, increasingly, colleges achieve, not what resources they 
receive, that matters most. 


Leading Learning and Policy; the New Professional Practitioner Paradigm 


Managing a modern school, or a school district, for academic achievement is a far different 
and more daunting undertaking than pursuing past paths of ensuring student control, parental 
acquiescence, and political stability. Similarly, seeking a parallel ability to measure and manage 
value-added in higher education is a challenge. 

The professional skills and understanding that drive achievement involve technical and pro- 
fessional complexities such as knowledge of human learning, understanding of curriculum ob- 
jectives, familiarity with modern performance measurement, comfort with managing by data, a 
feel for organizational complexity, conscious attention to leadership dynamics, appreciation for 
policy imperatives, comprehension of legal underpinnings, strategies for community and political 
engagement, understanding of budgetary processes and the empowerment they entail, integration 
of technology into both management and instruction, and a sustained sense of the significance of 
public information, community relations, and the ever evolving policy context. 

If properly conveyed, the above practitioner-oriented curricular components and professional 
experiences fill a doctoral program, all by themselves. It is unrealistic to think that one can layer 
significant research preparation on top, or alongside, of the above-listed complexity. 

Conveying modern research capacity is equally complicated. Leadership and policy research 
training are not functions of coursework alone. To be sure, there are fundamentals of research 
logic and design and multivariate statistics. But this is but a fraction of what producing a modern 
researcher entails. There are two other significant factors. One is substantive knowledge of 
a cognate field, be it psychology, political science, sociology, or economics. The other, and the 
most time-consuming of all, is immersion in the act of research and the intense personal mentoring 
such should involve. 

No self-respecting doctoral program attuned to the production of modern education researchers 
can possibly layer multidisciplinary cognate knowledge, understanding of education institutions, 
research immersion, data set understanding, comprehension of methods, and mentoring on top 
of a full professional curriculum and expect to cover the content in less than 7 years. It certainly 
cannot be undertaken in the context of a part-time doctoral program. 
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WHEN DID EDUCATION RESEARCH BECOME SO SOPHISTICATED? 


No Child Left Behind symbolizes the transformation of education leadership expectations. What 
is the research analog? 

How is education research being changed from a cottage industry dominated by single-shot 
case studies, interviews, literature reviews, and mail surveys, undertaken by doctoral students 
and education school faculty members, into a modern multidisciplinary community of social and 
behavioral scientists and skilled analysts embracing and promoting experimentation and research 
rigor? 

Once education or schooling itself became significant for the majority of America’s students, it 
was inevitable that education research eventually would also undergo a paradigm shift. If elevating 
achievement, rather than maintaining the social and instructional status quo, is a schooling 
objective, then the quest for fundamental knowledge regarding learning, teaching, management, 
and policy also becomes elevated in significance. 

Thus, the surprise is not so much that education research is, at last, assuming a trajectory of 
scientific rigor but, rather, how fast the transition is taking place. Indeed, the intensification of 
research rigor is occurring at such a blinding pace that a casual observer cannot be faulted for 
missing the signal conditions that mark the transition. 

The abrupt transition to research rigor also has its roots in the No Child Left Behind Act. Here 
there are literally hundreds of references to the necessity of underpinning education practices and 
decisions upon scientific evidence. 

With NCLB as a precursor, congress passed the Education Sciences Institute Act in 2002. This 
legislation transformed a previously fuzzy concept of the federal government’s research role into 
a far more precise vision. It also set the stage for appointment of Grover (Russell) Whitehurst as 
the first commissioner of the Institute of Education Sciences (IES). 

In a remarkably short period of time (2003 to 2009), Whitehurst, a psychologist by training 
and a SUNY Stony Brook Professor by experience, transformed the federal research role. His 
insistence upon rigorous peer review; deployment of highly trained social scientists from eco- 
nomics, sociology, and statistics onto proposal review panels; rewriting of application guidelines 
to expect many research funding proposals be grounded in randomized field trials; outsourcing 
and Internet orientation of proposal submission processes; and injection of research expectations 
into previously moribund activities and fiscal sumps such as the Regional Educational Laborato- 
ries resulted in a seldom before matched transformation of a federal agency. Instead of following 
the pack, IES now is shaping American education research. 


CONCLUSION 


Given the new twin realities—rigorous expectations for both education research and practice—a 
single, all-in-one, advanced degree no longer is sensible. 

In many higher education institutions, those offering only a Ph.D. or only an Ed.D., or even 
in institutions in which both degrees are offered but little distinction is made between the two, a 
reform is needed. 
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The Ph.D. (historically research degrees, at least in most academic fields) and the Ed.D. (a 
practitioner degree, at least by initial intention) need to be separated, and each deserves a distinct 
purpose, program standing, and pride. 

Bringing such a separation into reality, establishing an individual dignity for each, is seldom 
easy. Such a separation is as delicate and dangerous to the overall organization as is the surgery 
necessary to separate human conjoined twins. 

It can be done. A select few institutions have succeeded. However, the steps necessary for 
doing so successfully are complicated in nature and intense in their time commitment to complete. 

Specifying and explaining these steps is the subject for this issue’s subsequent series of articles. 
For now, suffice it to argue that such a separation of doctoral degree purposes is now imperative 
for professional pride, institutional regard, and individual student participant well-being. 

There is a final consideration. Those who contend that one degree is the solution or is better 
than two degrees often rely upon the prestige argument. They contend that the Ed.D. is tainted and 
only reliance upon the Ph.D. offers a prospect that the degree will convey status and be perceived 
with regard. 

Though this argument is not foolish, it is short sighted. As soon as a non-educator is made 
aware that the Ph.D. at hand was awarded in education, the recipient’s status is at risk. 

The solution is not in a single or even a dual degree. The ultimate answer is to render either or 
both of such rigor as to ensure a deserved level of high regard. 
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The Changing Context of K-12 Education Administration: 
Consequences for Ed.D. Program Design and Delivery 


Ellen Goldring and Patrick Schuermann 
Peabody College of Vanderbilt University 


This article explores the changing context of school leadership in our nation, a context that requires 
education leaders who are skilled and knowledgeable with a new set of dispositions to lead complex, 
diverse, and innovative institutions. The article also discusses recent critiques of existing leadership 
preparation programs, with emphasis on what has been said about the Ed.D. and Ph.D. degrees 
in educational leadership. Data are provided about trends in advanced degree offerings in higher 
education institutions, and the mission statements of top-ranked graduate programs in education 
leadership are assessed. Finally, the article discusses the consequences that these trends, critiques, 
and the changing context of school leadership might have upon the design and delivery of leadership 
preparation programs for current and aspiring school leaders. 


INTRODUCTION 


Spurred on by writings such at Thomas Friedman’s The World Is Flat and multimedia messages 
such as “Shift Happens,”! the evidence of global change is becoming increasingly elevated in 
our nation’s conscience. These works illuminate the great demographic, social, technological, 
and economic shifts that are altering the landscape of our world, a world that is demanding 
more from our nation’s schools and school leaders than at any point in our history. These forces 
have heightened our expectations for student performance and influence our vision of effective 
education leadership. Clearly, contemporary school leaders face a daunting array of challenges, 
are called upon to serve an evolving range of roles, and must draw upon a breadth of knowledge 
and skills to provide effective leadership to the students, teachers, and communities whom they 
serve. This article explores the changing context of school leadership in our nation, summarizes 
recent critiques of leadership preparation programs, and discusses the consequences of these 
activities for Ed.D. program design and delivery. 
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THE CHANGING CONTEXT OF SCHOOL LEADERSHIP 


Much has been written about the changing nature of school leadership in recent years (see Murphy, 
Hawley, & Young, 2005). Those factors that have had the most influence on altering the field of 
educational administration have raised our awareness about the need to consider a redefinition 
of the Ed.D. These factors are related to a changing context for school leadership and require 
educational leaders who are skilled and knowledgeable with a new set of dispositions to lead far 
more complex, diverse, and innovative institutions. The changing context includes: 


e Enhanced accountability demands 

e A learner-centered leadership focus 

e The necessity of analytic skills: Gathering research evidence and data-based decision- 
making 

e The influence of competition and school choice 

e Expectations for system-wide community engagement 


New Accountability Demands 


Perhaps one of the biggest shifts in expectation facing educational leaders is the notion of account- 
ability. Schools are expected to achieve results for all students in terms of student achievement 
and high school graduation. Although much of the focus in recent years has been on external 
accountability as defined in the No Child Left Behind legislation, accountability stems from both 
external and internal accountability systems (Adams & Kirst, 1999). External accountability 
refers to performance expectations that emerge from outside the school and the local community, 
whereas internal accountability refers to local goals and expectations for the school system and/or 
individual schools. 

Accountability entails monitoring, challenging, and rewarding. In today’s educational land- 
scape, external accountability expectations result in the public dissemination of data. Public 
knowledge comprises some of the most widespread incentives and pressures for educators to 
improve their students’ performance. For example, in many states students must pass high school 
tests to graduate; in other states, tests scores may be tied to pay raises. The stakes associated with 
such test results are high, and they may only go higher. 

The increased emphasis on accountability has numerous implications for the role of educational 
leadership and therefore the training and preparation needed by school leaders. Accountability 
has increased the visibility and responsibility of educational leaders; no longer is it possible for 
leaders to maintain or merely manage educational systems or focus on non-educational outcomes, 
such as facilities and transportation. Educational leaders must now align school systems with the 
achievement of clear, measurable goals and standards related to student learning. Educational 
leaders must play a key role in articulating and setting standards, developing and understand- 
ing measures of accountability, and aligning local expectations and accountability with external 
frameworks. In addition, leaders need to engage teachers and other stakeholders in meaningful, 
collaborative discourse around standards and measures while providing professional development 
opportunities and other supports to help them meet established goals (Newmann, 1997). Educa- 
tional leaders must hold their staff accountable for implementing strategies that align teaching 
and learning with achievement goals and targets set by policy. 


THE CHANGING CONTEXT OF K-12 EDUCATION ADMINISTRATION ial 


As noted by Murphy, Elliott, Goldring, and Porter (2007), school leaders play an integral role 
in focusing their staff and students on the particular criteria for success embodied in performance 
standards and school goals. They do so through frequent reference to and use of these criteria in 
performance reviews, classroom observations, discussions of curriculum and instructional strate- 
gies, and other interactions with staff. Effective leaders help ensure that school communities meet 
and exceed accountability expectations by offering individualized support to staff, challenging 
teachers to think critically about their teaching, and promoting an atmosphere of collaboration 
and communication. 


Learner-Centered Leadership Focus 


Perhaps because of the increased press for accountability for student learning, there is a renewed 
focus on instructional improvement as a path toward meeting standards. This focus requires 
leaders at all levels of the system to have a deep understanding of and engagement with teaching 
and learning and a strong grasp and awareness of the subsystems of the educational system that 
are necessary to support teaching, learning, and achievement. 

Research documents that schools with shared, focused visions about instruction, an aligned 
and rigorous curriculum, norms of collaboration, collective responsibility for students’ academic 
success, and an ongoing reflective dialogue among staff about practice create incentives and 
opportunities for improving teachers’ practices (Bryk & Driscoll, 1985; Hallinger & Heck, 1996; 
Newman & Wehlage, 1995; Purkey & Smith, 1983). Sustained coherence and alignment among 
resources, learning communities, and external constituencies are also essential components of 
school improvement (Knapp, Copland, & Talbert, 2003). An emphasis on learning-centered 
leadership is helpful for the development and maintenance of these conditions. Unfortunately, 
many school leaders are ill prepared to create and sustain these conditions (Goldring & Cohen- 
Vogel, 2000). The lack of preparation is even more salient because the nature of instruction is 
changing to ensure that all children can learn. Unless school leaders practice in ways that support 
teacher learning, recent federal and state reform initiatives are likely to have, at best, a marginal 
effect on teaching and learning. 

Since the effective schools movement, radically differing views of teaching and learning have 
emerged. Educational reformers advocate altering teaching and learning to take into account how 
students learn, good pedagogical practice, and the nature of the content or discipline (Stevenson, 
1996). New models of teaching and learning are placing greater emphasis on content, subject 
matter, and authentic pedagogies. Simultaneously, state-level curricula frameworks, accompanied 
by high-stakes testing and school accountability expectations, are placing educational leaders at 
the center of instructional reform efforts. This new emphasis on teaching, learning, curriculum, 
assessment, and pedagogy requires that educational leaders, and therefore Ed.D. curricula, address 
domains that have been largely left out of the knowledge base for educational leaders (Hess & 
Kelly, 2005; Levine, 2005). 


New Analytic Skills: Gathering Research Evidence and Data-Based 
Decision-Making 


Scientific research, or evidenced-based research, and data-based decision-making are now core 
technologies and skills for educational leaders. No Child Left Behind requires that programs 
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and practices implemented in school systems are grounded in scientifically based research that 
is research grounded in sound evidence. Furthermore, because of the testing requirements in No 
Child Left Behind, states, districts, and schools and their leaders have access to multiple types 
and sources of data and are faced with the challenge of how to use new analytic tools to lead their 
organizations. 2 

Though using data to respond to and monitor internal and external accountability is necessary, 
data-based decision-making is also a powerful tool to drive strategic planning and resource 
allocation decisions. A wealth of data is available that can be used to evaluate polices and 
programs and develop and improve curriculum, teaching, and learning. In this case, data are 
used to guide instructional improvement; they are not an end-of-the-year occurrence. Data are 
lynchpins in planning and implementing focused improvement strategies. Multiple types and 
sources of data are used to understand the educational system’s strengths and weaknesses, set 
priorities, focus change efforts, and establish a baseline from which to monitor progress. 

School leaders can use data to develop a culture of learning for adults as well as students 
in the system with the regular flow of information from data. Leaders can use data to put 
into place mechanisms to support individual and collective learning surrounding data. Teams of 
professionals, teachers, and state-level department heads, for example, have access to information 
that is used to build collaborative relationships and drive cycles of inquiry. These teams are guided 
by questions such as: What are the data telling us? What can we learn from the data? How can we 
change our practice in light of the data? What other data do we need to collect? Certainly, data that 
are dispersed and shared throughout the system can facilitate enhancements in the professional 
climate and spur organizational learning. 

Outside of schools, standards and greater use of data offer an opportunity to bridge the di- 
vide that often exists between schools, the community, and parents. As schools are required to 
communicate more information about their students to the community at large, deeper dialogue 
between educators and parents can also occur. This school-community relationship works in 
both directions: parents can understand more about their students’ academic achievements, and 
educators can gain a broader understanding of their students’ backgrounds. In a recent evaluation 
of the Children Achieving school program in Philadelphia public schools, the Consortium for 
Policy Research in Education documented how the program included parents in setting learn- 
ing standards to evaluate student progress in different subject areas. Parents reported a deeper 
understanding and appreciation for the curricula and a greater sense of inclusion in the school. 
Educators, for their part, described an appreciation for the contributions many parents made to 
the process (Golds, Rhodes, Brown, Lytle, & Waff, 2001). In addition to the increased communi- 
cation that comes through publicizing certain data such as standardized tests, educators can use 
data to encourage parental involvement in their students’ educations. 

The implications of the shift from making decisions based on intuition, anecdotal evidence, or 
experience toward decisions based on research and data for educational leaders are profound. On 
multiple topics, research is emerging that addresses questions about what works and under what 
condition specific strategies or interventions may work. This research can serve as an important 
knowledge base for determining how to address problems identified by the data-based decision- 
making processes. Educational leaders need the skills to lead with data and the knowledge to 
interpret and use complex research. If not already, the development of these capacities should 
become integral components of leadership preparation programs. 
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Increasing Competition and School Choice 


If in the past, school leaders could be guaranteed student enrollment stability because of strict 
attendance zones and limited school choice, this is no longer the case. When examining school 
choice, one is struck by the variety of options available in the United States, such as school 
vouchers, charter schools, magnet schools, open enrollment, private schools, homeschooling, 
and school transfer options under No Child Left Behind. There is a growing body of research 
on whether these various options are related to student achievement, and the results of this 
research are mixed—whether considering the effects of charter schools or magnet schools or 
vouchers to attend parochial schools (Betts & Hill, 2006; Betts & Loveless, 2005; Gill, Timpane, 
Ross, & Brewer, 2001). Nonetheless, educational leaders are now faced with heightened market 
accountability that is derived, in part, from elevated competition and differentiation in school 
choices. 

Most school choice plans, even those in the public school arena, are based on a market 
orientation toward school improvement. It is assumed that parents will choose schools, and 
schools will exhibit some level of responsiveness to meet the various needs and preferences of 
their “clients” (i.e., students and their parents; Chubb & Moe, 1990). According to the theory 
of voice and exit, if schools fail to respond to parental dissatisfaction, parents will enroll their 
children elsewhere (Hirschman, 1970). Following these lines of study, schools will improve to 
compete to attract parents and students. 

Public school choice has been given new prominence under No Child Left Behind, which 
requires that students in failing schools be given the option to transfer to more successful schools 
in their districts. This is in keeping with other provisions of the act that seek to strengthen 
accountability in public education. The prospect of losing students is meant to operate as a 
sanction to spur failing schools to improve. However, this is not the only purpose served by 
this provision of the law. Advocates of expanded choice in public education do not see choice 
merely as a sanction. Choice is also viewed as a mechanism for creating more successful schools, 
particularly in communities where low income makes it difficult for parents to exercise options 
available to the more affluent, who choose better schools by changing residential location (see 
Henig, 1999). 

Chubb and Moe (1990) and others (e.g., M. Friedman, 1962; Kearns & Doyle, 1988; 
Lieberman, 1989, 1993) argue that allowing parents to choose their children’s schools will result 
in market-like competition and the decline of bureaucratic structures, thus providing parents with 
greater opportunities for home-school interaction and a greater openness on the part of schools 
to parents’ demands. Supporters of de-bureaucratization claim that, in such a system, parents, 
especially low-income and minority parents, will be less intimidated by the school and more 
willing to make their needs known to school personnel (e.g., Cookson, 1994; Rinehart & Lee, 
1991). 

Perhaps one of the most striking changes in schools of choice is the nature of the interrela- 
tionships between parents and the schools’ their children attend. Schools of choice may provide 
incentives and opportunities for parents and teachers to engage in positive home-school inter- 
actions for several reasons. Three mechanisms may explain the way charter schools and other 
schools of choice may support higher levels of parent involvement: the act of choice itself, the 
school—parent fit or match, and the nature of school organization and institutional environments. 


14 E. GOLDRING AND P. SCHUERMANN 


First, parents normally make a conscious decision to choose a particular school, although it may 
be only a single alternative to the neighboring public school. In so doing, there is the possibility 
for greater cooperation between the school and families in the education of the student. The act 
of being selected by the school to attend, and the decision by the family to enroll, assumes at 
least the basis of a partnership, or community. Thus, under choice arrangements, teachers may be 
more open to working with students and their families and families more open to working with 
the school. 

Second, the focused program or purpose of many schools of choice may increase the possi- 
bility of greater congruence between what families want for their children and what the school 
purports to provide, thereby proposing a good fit between home and school. Earlier literature 
on effective schools (e.g., Brookover, 1979; Goldring & Shapira, 1993) and literature on re- 
structuring emphasizes the need for goal consensus (e.g., Elmore, 1990). School choice attempts 
to resolve conflicts over goals through voluntary membership (Bryk, Lee, & Holland, 1993; 
Metz, 1990; Raywid, 1985). Bifulco and Ladd (2005) talk about the “parent-school match” 
in charter schools where parents could feel a heightened comfort level and willingness to be 
involved. 

Third, schools of choice may embody certain communal-organizational characteristics that 
will enhance home-school relationships. Schools of choice are thought to consist of decentralized 
mechanisms of control, enabling them to operate less bureaucratically than traditionally organized 
public schools (e.g., Chubb & Moe, 1990; Raywid, 1985). Public schools of choice are often 
exempt from some rules that generally govern other public schools, emphasize greater teacher 
decision-making and collegiality, develop a curriculum focus, and admit students through a 
screening process based on parent choice and the students’ interest or suitability for the program 
offered. 

The implications of increased competition and choice for education leaders are widespread. 
Parents have new and different expectations of teachers, principals, and the educational system 
as a whole. They may be seeking a more flexible, personable, and non-bureaucratic approach to 
education. Leaders are responsible for crafting a compelling vision and strategies to implement 
and reach goals, not only for parents who have choices but for educators. With increased choice, 
both teachers and principals have options as to where they would like to work. Educational 
leaders now have to compete in the labor market within the educational sector. Thus, market 
accountability also requires that educational leaders create working conditions that attract the 
most highly qualified educators. 


System-Wide Community Integration 


In the past, educational leaders were primarily responsible only to the school system. However, 
a new “open-system” perspective, the imperative of school achievement for all, and increased 
accountability and choice have repositioned educational leadership to be part of a larger, interre- 
lated community context. This context includes interactions between the educational system and 
its leaders and other communal institutions, such as the juvenile court system, the police depart- 
ment, welfare and health services, the business community, and private and public philanthropic 
foundations. 
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This view of educational institutions emerges from the notion that public schools can be tools 
for community and economic development. A major report by the Harvard University Joint Center 
on Housing Studies stated: 


As a major place-based infrastructure element and integral part of the community fabric, public schools 
can have a profound impact on the social, economic and psychological character of a neighborhood. 
Addressing public schools, therefore, is a good point of entry for community based developers to 
place their work in a comprehensive community development context. (Chung, 2002, p. iii) 


In addition, integrating school reform and neighborhood revitalization is central to community 
development initiatives. This perspective was behind school reform programs in many of the 
most challenged schools and communities such as Baltimore, St. Louis, and Atlanta (Proscio, 
2004). In the public school arena, community development with schools often takes on such 
projects as providing better school-to-work transitions by developing better relationships with 
local employers and creating schools that are joint-use facilities, such as adult job training centers 
in the evenings. 

Schools are now required to be much more integrated with other non-educational institutions in 
the community, thereby embodying a “holistic” perspective (McGaughey, 2001, p. 125). Timpane 
and Reich (1997) call this “an ecosystem—a total environment supporting the healthy growth 
and development of America’s youth” (pp. 465-466). Often referred to as the “alliance” model, 
existing institutions within a community—of which schools are often the most prominent— 
work together to harness the resources available to the community at large (Crowson, 2003). 
Informal networks and arrangements together with formal structures help to provide support for 
the development of institutions within a community. 

This perspective has major implications for educational leadership: 


At its heart, community development constitutes a philosophical change in the way schooling is 
conceived. Community development changes the core identity of schools from isolated, independent 
agencies to institutions enmeshed with other community agencies in an interconnected landscape of 
support for the well-being of students and learners. . . . It beckons schools to consider and respond to 
learning needs throughout the community, not just to those of children within the school building. 
(Timpane & Reich, 1997, p. 466) 


With a greater emphasis on community-wide engagement, there is a necessary redefinition of 
roles, skills, and dispositions of school leaders to include communication and political positioning 
beyond the school house and boards of education. Recently, the concept of civic capacity has 
been applied to the notion of community development and school reform (Orr, 1996). Civic 
capacity is defined as the “degree to which a cross-sector coalition comes together in support of 
a task of community-wide importance” (Stone, 1998, p. 234). The concept refers to an ability to 
build and maintain effective alliances for collective problem solving (Orr, 1996). Civic capacity 
emphasizes the collective role of community stakeholders, going beyond the view that any one 
single institution, such as schools, can address the needs of its constituencies. “In the education 
arena, the capacity of any set of stakeholders is limited, and so long as various players think and 
act in terms of a narrow view of their duty, they miss the full scope of the problem they face and 
the response to which they could contribute” (Stone, 1998, p. 254). 
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Community engagement will not happen on its own; people from various agencies, institu- 
tions, and groups do not naturally come together to solve mutual problems and develop initia- 
tives. Therefore, today’s educational leaders need to motivate community-mindedness to address 
community-wide problems that are central to schools and the current imperatives of student 
achievement. ; 

Taken together, the five factors described above represent the ways in which the expectations 
for and the context of school leadership are changing. These changes pose very real challenges to 
school leaders, as managing a school, or school district, within the current educational context is 
far different than in the past. New professional skills, instructional understandings, and leadership 
competencies are necessary to drive student achievement in the complex, ever changing, and 
increasingly innovative context within which our schools are situated. These forces illustrate the 
importance of reconsidering how school leaders are prepared for modern-day challenges within 
advanced degree programs in our nation, most notably the Ed.D. program. 


The Need for a Refocused Ed.D Degree 


If our educational institutions were subject to the same environmental forces that move species 
along a trajectory of survival of the fittest, what would our leadership preparation programs 
look like today? If the field of education was subject to the same professional scrutiny and 
confronted squarely the corollary consequences that mark the fields of medicine and law, would 
school and district leaders be trained any differently? The global shifts impacting our nation, 
the changing context of school leadership described above, and the growing sophistication of 
schooling and education research illuminated by James Guthrie (2009) in this issue each have 
significant consequences for Ed.D. programs. In this section of the article the authors review 
the background analyses that dominated the field in terms of calling for a new Ed.D degree 
and separating the Ph.D. and the Ed.D. Then, we focus upon the manner in which the changing 
context of school leadership can inform thinking about leadership preparation and discuss the 
consequences that the evolving role of school leaders might have upon Ed.D. program design and 
delivery. 


Summary of Educating School Leaders by Arthur Levine and Reclaiming 
Education's Doctorates by Shulman et al. 


In 2005, Arthur Levine released a report titled Educating School Leaders in an effort to assess the 
state of school administration graduate programs in departments and colleges of education across 
the country. In this report, Levine synthesizes results from national surveys of the following 
four groups: education college deans and education department chairs, education school faculty 
members, program alumni, and school principals. Case studies of 28 university departments or 
schools of education, selected to reflect diversity of location, demographics, and Carnegie type, 
were also included. 

The report aims to consider how well education administration and leadership programs 
prepare today’s principals for the jobs that await them. Levine devised a framework by which 
to judge school leadership programs. The template included nine points of assessment outlined 
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as follows: (a) Purpose: The purpose of the program is clearly stated and is relevant to today’s 
requirements of principals; (b) curricular coherence: The curriculum is rigorous and aligned 
with the stated purpose of the program; (c) curricular balance: Both theory and practice are 
emphasized in the program; (d) faculty composition: The faculty is balanced with both academics 
and practitioners. Faculty members are in touch with current issues facing school principals and 
schools, in addition to theory; (e) admissions: Admissions criteria are sufficiently high and are 
designed to recruit capable, driven students; (f) degrees: Graduation standards are high and degrees 
are aligned to the profession; (g) research: Research conducted in the program is distinguished 
and relevant to practice, not just theory; (h) finances: The program has sufficient resources; and 
finally, (i) assessment: Evaluation of the program is ongoing within the school or department of 
education. 

When assessed by Levine according to this framework, the nation’s education administration 
programs were overwhelmingly disappointing. None were found to satisfy all nine of the criteria. 
Two programs, those at University of Wisconsin—Madison and Peabody College of Vanderbilt 
University, were found to meet most of the criteria. 

In this analysis, four observations were especially worrisome to the author. First, an increas- 
ing number of institutions have developed off-campus satellites at which educational leadership 
programs are among the courses offered. Programs at such regional campuses are often charac- 
terized by low-quality courses and faculties composed primarily of adjunct professors. Secondly, 
in an effort to improve institutional status, some universities (mostly some master’s I and weaker 
research-intensive institutions) created doctoral programs in school administration despite their 
lack of resources. The report labels them “credit dispensers” and notes that the degrees awarded 
are too often in name only (Levine, 2005, p. 24). The third phenomenon is the “race to the bottom” 
that occurs when schools and departments lower admission standards and program expectations 
in an effort to attract students from their institutional competitors. The author contends that uni- 
versities are looking to “produce more degrees faster, easier, and more cheaply” (Levine, 2005, 
p. 24). Finally, Levine observed that state and district policies that award salary raises for those 
who complete an advanced degree only encouraged this student culture of looking for ease and 
rapid graduation. 

Levine’s report categorized the main issues of current school leadership programs into six 
overarching problem areas: irrelevant curriculum, low admission and graduation standards, weak 
faculty, inadequate clinical instruction, inappropriate degrees, and poor research. The first, irrel- 
evant curriculum, addresses the finding that program curricula are out of touch with the needs of 
today’s principals. When surveyed, “an average of only 63 percent of principals found the courses 
valuable” (Levine, 2005, p. 28). Quality and rigor were also questioned. 

Low admission and graduation standards are the second problem area noted by Levine. 
Education leadership programs tended to have the lowest admission standards of any graduate 
program offered in education departments and colleges. Alumni survey responses emphasized 
student interest in convenience over quality. Additionally, many school leadership students, if not 
most, balance a full-time job with full-time school, a juggling act that often takes away from the 
student’s academic investment. 

Third on Levine’s (2005) list of problems is a weak faculty, which he attributes to the following 
paradox: “On the one hand, the field depends too heavily on practitioners serving as part-time 
faculty, and on the other, it employs too many full-time professors who have minimal, if any, recent 
experience in the practice of school administration” (p. 35). Part-time adjunct professors typically 
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lack the deep expertise in theory and research desired, whereas academic faculty members lack 
a connection to practice. 

Fourth on the list is inadequate clinical instruction, referring to the less-than-rich internships 
or practicum experiences that many programs require of students. Levine (2005) notes: “For the 
most part, students described the experience as something to be gotten out of the way, not as a 
learning opportunity” (p. 40). Too often connections were not drawn between academic lessons 
and hands-on experience. 

The fifth problem highlighted is the misaligned array of degrees and purposes. The disagree- 
ment in the field about the purposes and characteristics of school leadership programs is made 
visible by the great diversity of degrees; for example, the line between the Ed.D. and Ph.D. is 
often hazy. Levine (2005) writes: “The problem is that so many practitioners are working toward 
a degree that was intended to prepare academic researchers and scholars and that has no relevance 
to their jobs. ... As a consequence, professors at research institutions find themselves having to 
compromise on the quality of the dissertation” (p. 43). More will be said about the debate between 
the Ed.D and the Ph.D. in the next section. 

Levine’s (2005) final problem diagnosis is the lack of both rigorous and relevant research. Sur- 
vey responses criticized research for “confusing scholarly and practical inquiry, flitting from 
topic to topic, prizing breadth over depth, and being abstruse,” in addition to “its overde- 
pendence on qualitative methodologies” (p. 44). Policy-makers and practitioners essentially 
ignore academic publications and instead turn to trade papers and professional association 
publications. 

The report offers a reform framework buttressed by several specific recommendations. The 
first part of the framework involves efforts to “eliminate incentives that promote low quality 
programs” (Levine, 2005, p. 63). To make this happen, Levine first recommends that school 
districts and states end policies that give salary increases for advanced credits and degrees. He 
suggests that, instead, emphasis should be given to mastery of skills and knowledge, rather than 
ascertaining a meaningless degree. Salary incentives should only be offered for courses that 
are directly relevant to an individual’s position. Another recommendation of the report is that 
education schools and departments assess their financial structures and work with the university 
to improve resources. Rather than allowing the education schools to serve as “cash cows” for the 
university at large, education schools need to actively work to redirect the flow of finances inward 
to adequately support school leadership programs. Sufficient resources would allow education 
schools to feel less pressure to accept all (or virtually all) that apply and focus again on sufficiently 
high standards. 

The second component of the reform framework is to “set and enforce minimum standards 
of quality” (Levine, 2005, p. 65). This strategy can be carried about by Levine’s third recom- 
mendation: “weak programs should be strengthened or closed” (p. 65). Though quality control 
is the responsibility of school leadership program leaders, should they fail to act responsibly, 
the author asserts that universities should take charge. Should universities also fail to act, states 
should realize the grave situation and take ultimate responsibility. 

Levine’s final reform strategy suggests that education administration programs be redesigned. 
Three recommendations are provided to carry this strategy out. First, in an effort to align appro- 
priate skills and knowledge to the jobs of school leaders, education leadership degrees should 
parallel the M.B.A. in the business sector. Curricula should incorporate courses in both man- 
agement and education. Theory and practice would both be relevant aspects of the curriculum, 
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and faculty would reflect this purpose. Secondly, the doctorate of education (Ed.D.) should be 
eliminated altogether. The doctorate is unnecessary for practitioners such as principals and su- 
perintendents, and any future researchers should instead be served by the Ph.D. This leads to the 
third recommendation. The doctorate of philosophy in education (Ph.D.) should be redefined and 
reserved only for those wanting to pursue research and become career faculty in the traditional 
sense. Ph.D. program sizes would greatly shrink, allowing for an increase in rigor and realignment 
with research and theory. 

Writing a year after Levine, Shulman, Golde, Bueschel, and Garabedian (2006) also weigh in 
on the current state of school leadership programs and the misalignment of degrees in “Reclaiming 
Education’s Doctorates: A Critique and a Proposal.” They write: “Unlike other fields, ours uses 
the doctorate both to prepare scholars and to prepare the highest level of leading practitioners 
(McClintock, 2005). Often, through confusion of purpose, we end by doing neither very well” 
(Shulman et al., 2006, p. 26). 

Shulman et al. (2006) contend that the problem is that Ed.D. and Ph.D. programs are not 
aligned to their theoretically separate purposes. In theory, the Ed.D. degree programs are intended 
to train practitioners “for managerial and administrative leadership,’ whereas Ph.D. programs are 
reserved to prepare researchers and academics (p. 26). Instead, poorly structured programs and 
confusion of purpose have caused the Ed.D. to be viewed as the “Ph.D. Lite,” rather than as a 
separate degree for a separate profession. By comparing education graduate programs to those 
in the arts and sciences, the authors also make note of the atypical nature of education doctoral 
students. Most students enroll in graduate study mid-career and, consequently, often balance 
school while continuing full-time work in the field. All of these characteristics make defining 
programs difficult. The article highlights three institutions, University of Colorado at Boulder, 
University of Southern California, and University of North Carolina at Chapel Hill, to demonstrate 
program issues and examples of reform. 

Unlike Levine’s recommendation of an M.B.A. equivalent in educational leadership, Shulman 
et al. (2006) believe practitioners in the field require doctoral study. Shulman’s report proposes 
a new degree, the “Professional Practice Doctorate” (P.P.D.). Though the name is of little im- 
portance, they emphasize the need to start with a clean slate, rather than trying to revise the 
already confusing Ed.D. The goal is to create this new doctorate reserved solely for practitioners, 
while slowly returning the Ph.D. to a small group of education researchers. Using the analogy 
of the distinction in medicine between the M.D. and Ph.D., Shulman et al. recommend a similar 
dichotomy of practice and research. Like the medical degrees, both doctorates would be highly 
rigorous and prestigious, and the differentiation between the two would be clear and unmistak- 
able. Each degree would be linked to a “signature pedagogy” (Shulman, 2005, as cited in Shulman 
et.al., 2006). 

Shulman et al. (2006) recommend shaping the P.P.D. first by the appropriate assessments. Just 
as assessments for the M.D. are more varied than the Ph.D. in medicine, assessments for the 
P.P.D. too should be diverse, determining individual ability to apply knowledge to practice. As for 
clinical instruction, the comparison is drawn to the medical residency. P.P.D. students would be 
required to be able to “read research reports, very critically and analytically,” (p. 24) but skilled 
training in how to conduct research would be reserved for the Ph.D. To address the part-time, 
mid-career nature of school leadership students, the authors suggest that P.P.D. programs be 
designed to incorporate work with study while requiring students to engage fully with academic 
life in their final year of the program. Shulman et al. acknowledge that revision of the Ed.D. 
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program is more likely than the development of a new PPD. but that as long as reform is made 
to realign the appropriate degree with its intended purpose, the degree name is unimportant. 


Critiques of the Levine and Shulman et al. Reports 


Quickly after the release of the Levine and Shulman et al. reports, school leadership faculty, 
education organizations, and foundations responded. Many agreed with several of Levine’s points, 
such as education schools must align degrees with high standards; school leadership programs 
can no longer be used as “cash cows” for universities, and financial resources must be secured for 
education programs; and there are many inadequate programs in the field (Pauls, 2005; Young, 
Crow, Orr, Ogawa, & Creighton, 2005). Balancing these endorsements were those who pointed 
out multiple weaknesses of Levine’s report. Among them, Michelle Young and colleagues (2005) 
and Leo W. Pauls (2005) separately note that Levine makes “sweeping generalizations” about 
the inadequate quality of school leadership programs. These authors highlight the importance of 
not lumping all programs together when there is such diversity in purpose, enrollment size, rigor, 
and certainly quality. A second line of criticism involved the report’s failure to acknowledge 
the reform efforts currently underway in leadership preparation programs. Young et al. (2005) 
reference work of the Interstate School Leaders Licensure Consortium, University Council for 
Educational Administration, and Taskforce on Evaluating Leadership Preparation Programs, 
whereas Pauls (2005) emphasizes the work of the Renaissance Group. A third line of critique 
involved Levine’s methodology, raising questions about the sample size, survey response rates, 
and the anecdotal interview data used to draw conclusions (Young et al., 2005). Finally, even 
when members of the field agree with his findings, many do not agree with his recommendations 
(Jacobson, 2005). Among Levine’s recommendations that are criticized are the belief that the 
Ed.D. degree should be eliminated and the report’s proposal to replace the doctorate degree with 
the masters degree for school administrators (Jacobson, 2005; Young et al., 2005). 

Levine’s report was not alone in being met with criticism. For example, Rodney Evans (2007) 
responded to the conclusions made by Shulman et al. and disagreed with the authors on a 
philosophical level, asserting that the article too simplistically sorted practice from theory. Evans 
is concerned that opinions in the field, such as those of Shulman et al., have ignored the importance 
of teaching doctorate students to engage in scholarly inquiry and to think about education for the 
sake of theoretical thought. As for steps toward reform, Evans (2007) notes three cautions. First, 
“spurious assessments based on existing practice to drive the curriculum-building process needs to 
be put on hold, if not jettisoned outright” (p. 555). Secondly, the curricula of these programs should 
be left to scholarly faculty who understand the necessity of combining research with practice. 
Third, the field needs to act with caution and consider possible “long-term consequences” before 
making any changes that separate practice from research. 


Two Degrees or Not Two Degrees 


As the above critiques point out, one primary decision that schools of education must make, and 
continue to reevaluate, is the type of degrees that will be granted to those wishing to purse doctoral 
studies in education. Figures 1 and 2 provide historical context regarding the types of doctoral 
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degrees that have been offered in education during the past forty years in our country. Figure 1 
presents data regarding the number of institutions that have offered Ed.D and Ph.D. degrees in 
education in our nation. Here we see steady growth in the number of institutions offering doctoral 
degrees in our country over the past 40 years, with a 24% increase from 1979 to 1989 and an 
additional 21% increase between 1989 and 2008. 

As Figure 2 points out, though the overall growth in the number of doctoral degrees in 
education have increased during the past 40 years, Ed.D. and Ph.D. degree offerings have not 
followed parallel trajectories. In the past, nearly half of the institutions offering doctoral degrees 
in education offered only the Ph.D. degree, with another third of the institutions offering both 
Ph.D. and Ed. D. degrees. That is not the case today. Presently, a mere 18% offer only a Ph.D., 
with a third offering only an Ed.D. degree and a full half of the doctoral granting institutions in 
our nation offer both Ph.D. and Ed.D. degrees. 

Three conceptual approaches to resolving the issue of which advanced degrees to offer in 
education include the professional school argument, the autonomy argument, and the unification 
argument.” The professional school argument contends that because schools of education are 
professional schools, they should confer their own degree (the Ed.D.), much like schools of law, 
business, and medicine. The autonomy argument further endorses this stance and claims that by 
offering the Ed.D., schools of education can enjoy greater freedom from institutional regulations 
associated with the Ph.D., which is historically an arts and sciences degree. Finally, the unification 
argument asserts that schools of education are often overly diversified and fragmented, and a focus 
on the Ed.D. degree would bring greater clarity of purpose and unity of vision. 

Consideration of the changing context of school leadership has different implications for 
the three arguments outlined above. The growing complexity of educational leadership merits a 
program that is tailored to the preparation of leaders who are able to utilize a vast array of skills 
to provide sound educational experiences for those they serve. This trend would seem to play into 
the hand of the professional school argument. The autonomy argument is relatively impervious 
to contextual changes in educational leadership, as it remains almost exclusively responsive to 
organizational dynamics at the higher education level. The growing complexity of educational 
leadership would seem to nudge preparation programs in a direction of alignment and unification 
of programmatic elements, yet not to the exclusion of programs that would prepare researchers 
capable of technically rigorous empirical findings about the ever changing educational landscape. 

As outlined above, a transformation has altered the landscape of education leadership as ad- 
vances in educational research techniques and mounting complexities associated with leading 
educational institutions render a single track, dual purpose degree inadequate. In response to 
these realities, transformations are occurring across the nation in leadership preparation pro- 
grams. Even so, when assessing various degree program missions, methods, and requirements in 
education leadership, there is considerable variability. Analysis of the mission statements? of the 
top-ranked educational leadership Ed.D. and Ph.D.programs reveals this variability—both within 
similar degree programs and between the two degree types. 

The Ed.D and Ph.D. mission statements for Vanderbilt University’s Peabody College, the 
University of North Carolina—Chapel Hill, and the University of Utah all clearly distinguish 


2For a more robust treatment of this topic, readers are encouraged to see Osguthorpe and Wong (1993). “The Ph.D. 
Versus the Ed.D.: Time for a Decision.” 
3Mission statements are provided in the Appendix. 
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the Ed.D. program as one that prepares “practitioners to lead educational organizations” and the 
Ph.D. program as one designed for those who intend to build “an academic career focused on 
the generation of theoretical and research-based knowledge.” Though only offering an Ed.D. 
degree, the mission statement of Harvard University’s program emphasizes the development of 
both “researchers and practitioners,” those with an interest in the “development of knowledge” 
and the “application of research to practice.” 

Taking a different approach, the mission statement of the University of Wisconsin—Madison 
makes no reference to the context within which graduates will operate, by stating succinctly: “Our 
mission is to create, integrate, exchange, and apply knowledge about leadership, learning, and 
organizational performance to advance educational quality and opportunity.” Similarly, Pennsyl- 
vania State University seeks to “support and advance education by preparing outstanding leaders 
committed to the reform and continuous improvement of education.” 

The Ohio State University and Teachers College of Columbia University describe their Ph.D. 
programs as preparing individuals for “leadership and faculty roles in educational administration 
and to meet licensure requirements for administrative positions in schools” and for careers as 
“practitioners and scholars.” These Ph.D. program descriptions encompass both the research 
and the practitioner realms—goals that other schools might separate into two different degree 
programs. 

No matter which approach is taken to design doctoral programs in education, a distinguishing 
characteristic of advanced degrees is that context is central (Maxwell, 1996), and in recent years 
the contexts of education research and practice have grown increasingly distinct. 

The authors contend that at the advent of the 21st century, the technical capacities associated 
with education research and practice are now robust enough to overcome the inertia that allowed 
many institutions to offer both Ph.D. and Ed.D. degrees in an undifferentiated state. Increasing 
this demand are reviews by Levine (2005) and Shulman et al. (2006), which highlight historical 
and current deficiencies associated with preparation programs offered and degrees conferred in 
education. At the same time, and bolstered by such critiques, the field of leadership preparation in 
education is undertaking substantial self-assessment. The three leadership preparation programs 
highlighted later in this volume are exemplary of these reform efforts and are indicative of the 
outcomes emerging from this difficult work of appraisal, analysis, and action. Now, attention 
is turned to the implications of the changing context of education leadership on leadership 
preparation programs. 


CONTEXTUAL CHANGES AND CONSEQUENCES FOR ED.D. 
PROGRAMS 


The changing context of school leadership has important implications for the design and imple- 
mentation of Ed.D. programs. Heightened accountability demands, a shift toward learner-centered 
leadership, the necessity for elevated analytical skills, increasing competition and school choice, 
and the growing necessity for system-wide community engagement all require discussion and 
decision related to the content, delivery, program requirements, and purpose of Ed.D. programs. 
These consequences will be described by considering the manner in which the changing context 
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of school leadership can impact the why, what, how, where, and by whom of Ed.D. program 
design and delivery. 


The Why 


A primary way that the evolving context of school leadership has implications for Ed.D. programs 
is to encourage a thoughtful consideration of the core purpose and primary intent of the program. 
Initially, the Ed.D. was conceptualized for the purpose of developing scholarly practitioners 
who would enhance the field of education through applied research (Ringler & Rouse, 2007; 
Townsend, 2002). Whereas the prior discussion of mission statements highlights variability in the 
field, generally, Ed.D. programs are intended to develop professionals who will serve within the 
context of a school or district office. As highlighted earlier, schools of education will likely arrive 
at nuanced answers to this question based on various economic, social, or political demands. 
Even so, recent doctoral program critiques in education and the ever changing context of school 
leadership require continual analysis of the degree of alignment between program purpose, the 
demands of the marketplace, and community needs. 


The What 


If the development of expertise within context is critical, such as learner-centered leadership, 
then core programmatic components should be aligned with this purpose to develop this exper- 
tise. The curriculum that comprises an Ed.D. program should refine the judgment and enhance 
the decision-making capacity of school leaders. For example, recent research* highlights both 
core components of school leadership and the key process by which leaders operationalize and 
implement leadership. This view of leadership suggests that what is taught in Ed.D programs 
need not be confined to content areas but involve the interpersonal, social, and political dynam- 
ics that comprise the context of leadership. The factors that are driving changes in the roles of 
educational leaders highlight the need for a curriculum that addresses both the complexity of 
educational systems and new domains of learning in such areas as community engagement and 
data-based decision-making. 


The How 


Because the content expertise to be developed in Ed.D. programs is distinct from Ph.D. programs, 
new methodologies will need to be employed and pedagogy that takes into account the importance 
of context will be required (Maxwell & Shanahan, 1997). To address the complexity inherent in 
the context of professional practice, elements of action research, situated learning, and critical 
reflection will be important tools to help school leaders develop the analytic skills necessary for 
analysis of research evidence and data-based decision-making. 


4For a full description of this research, readers are encouraged to see Murphy, Goldring, Porter, and Elliot (2006). 
Learning Centered Leadership: A Conceptual Foundation. 
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Recently, Lee Shulman et al. (2006) posited that professions not only have specialized knowl- 
edge but also tend to have a “signature pedagogy,” which is a mode of teaching that has become 
closely linked with preparing people for a particular profession. A signature pedagogy serves 
many purposes because it is distinctive, pervasive within the curriculum, agreed upon across pro- 
grams and institutions, and serves as an element not only of instruction but of socialization into 
the profession. For example, in the law schools one would see the same pedagogy, namely, cases, 
regardless of the specific law school and across many courses in the study of law. Similarly, grand 
rounds are part of the pedagogy of medical schools, regardless of the specific medical school one 
attends. 

What could a signature pedagogy for the Ed.D. degree encompass? At a minimum, Ed.D. 
programs should incorporate some of the theories of learning that have considered how knowledge 
and expertise are developed and translated into usable knowledge. Knowledge that can be applied 
and transferred into the work context can provide some starting points for thinking about a 
signature pedagogy for the professional doctorate in education. 

Bransford, Brown, and Cocking (2003), in their seminal work How People Learn, suggest that 
learning should be knowledge centered, community centered, learner centered, and assessment 
centered. One direct implication is that the Ed.D. pedagogy should include problem-based cases 
that are situated in a context. Additionally, the pedagogy should provide opportunities for students 
to learn with and from a community of peers. For example, cohorts provide an opportunity for 
program participants to develop professional learning communities and thereby engage in the 
community-centered dimension of the How People Learn framework. To target the assessment- 
and knowledge-centered components of the framework, the Ed.D. pedagogy should provide 
many opportunities for ongoing feedback to participants and should allow students to explore 
evidenced-based research to answers for such questions as “What works, under what conditions, 
why and why not?” In addition, the Ed.D. pedagogy should be mindful of authentic summative 
evaluations of student knowledge and expertise. One approach to this dilemma is explained in 
detail by Smrekar and McGrainer (2009) in this volume when they describe the culminating 
Capstone Project of the Ed.D. program at Vanderbilt University’s Peabody College. 


The Where 


Almost exclusively, Ed.D. programs are designed and delivered on college campuses. Increas- 
ingly, distance learning is altering the physical location and proximity of instructors and course 
enrollees. Overwhelmingly, instructional units accrued in Ed.D. programs are still the product of 
hours spent in the classroom. In contrast to this, more programs are exploring alternatives, in- 
cluding internships and supervised clinical practice in institutional settings, that provide students 
opportunities to develop and apply skills and knowledge in context. Such work helps prospective 
leaders bridge theory and practice and provides opportunities to develop real-world expertise in 
such domains as engaging system-wide community support and implementing an intervention or 
change process into a school setting. In addition, the expert leaders and experienced practitioners 
are key to implementing these types of learning experiences where students receive ongoing 
feedback and coaching in response to their implementation efforts. Field experiences are no 
longer based on shadowing or hours but are geared toward accomplishing real institutional goals. 
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The By Whom 


One important take-away that emerges from a few decades of school reform research is that change 
initiatives, if they are to have any hope of sustained impact, must be crafted with key stakeholders. 
The same is true of successful attempts to enhance Ed.D. and ongoing leadership development 
programs. There exists a valuable synergy between university instructors, who bring highly 
developed analytic skills, and practitioners, who bring a high degree of knowledge in context 
(Maxwell & Shanahan, 1997). One consideration pertinent to this partnership is the balance 
in power relations between the two types of instructions and their knowledge. Here, Brennan 
(1996) encourages a democratization of professional authority whereby workplace knowledge is 
afforded equal legitimacy with analytic skill and academic knowledge. Toward this end, one must 
consider the extent to which decisions about Ed.D. program design and delivery have included 
the insights and voice of expert practitioners. One way to facilitate such collaboration is through 
the creation of an advisory board. For example, at Peabody College, the Ed.D. program benefits 
from a national board of advisors who provide input and feedback about many elements of the 
program. Later in this issue, Catherine Gavin Loss (2009) and Caboni and Proper (2009) discuss 
the role of this panel in more detail. 


CONCLUSION 


Changes in the educational landscape, alongside broader economic, social, and political shifts, 
have altered the context for K-12 education administration. Schools of education, associations, 
foundations, states, and professional organizations have responded to these conditions and are 
crafting a diverse array of leadership preparation programs that are strategically targeting lead- 
ership proficiency. By attending to ongoing alterations in the contextual landscape of school 
leadership and considering the consequences for Ed.D. program design and implementation, 
schools of education can ensure ongoing strategic alignment between their resources, the goals 
of their programs, and student needs. 
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Building, Sustaining, and Expanding the Education 
Doctorate at Peabody College: An Administrative View 


Catherine Gavin Loss 
Peabody College of Vanderbilt University 


This article examines the operational and administrative underpinnings of Vanderbilt University’s 
Ed.D. program. It explores the ways in which the admissions process, program orientation, and 
other academic and support services were reconceived when Peabody College formally launched 
its redesigned Ed.D. program in the summer of 2004. Further, it situates the mission of Peabody’s 
education doctorate—the training and preparation of practitioner-scholars in elementary, secondary, 
and postsecondary settings—within the context of program administration. 


Peabody College’s Department of Leadership, Policy and Organizations formally launched its 
redesigned Ed.D. program in the summer of 2004. The program revisions grew out of a desire to 
create a graduate experience that mirrored the Ph.D. program’s academic rigor and intensity but 
addressed the specific intellectual and professional needs of emerging school leaders. Distinct 
from its more traditional counterpart, Peabody College’s “new” Ed.D. program equips students 
with research skills and the broader theoretical perspectives that inform contemporary education 
but at the same time is anchored in the problems of practice. By integrating theory and practice, the 
Vanderbilt Ed.D. program is committed to the preparation of practitioner-scholars in elementary, 
secondary, and postsecondary settings. 

Revising the program presented intellectual as well as practical and administrative challenges. 
As the Ed.D. program moved from a residential, 9-month, semester format to a cohort-based 
weekend model designed exclusively for working professionals, the department was compelled 
to reexamine the administrative and operational structures needed to support and sustain such a 
transformation. The admissions process, program orientation, and other academic and support 
services were reconceived to meet the needs and demands of the program’s largely nonresiden- 
tial, working professional student population. The curriculum, capstone project, and Vanderbilt’s 
place within the national conversation on the education doctorate are detailed elsewhere in this 
special issue of the Peabody Journal of Education. This article takes a closer look at program ad- 
ministration, exploring some of the services and initiatives the Department of Leadership, Policy, 
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and Organizations has introduced to build, sustain, and expand its intellectual and professional 
community of Ed.D. students. 


BUILDING A COMMUNITY OF PRACTITIONER-SCHOLARS: 
ADMISSIONS, ORIENTATION, AND THE COHORT MODEL 


Each year, Peabody College welcomes approximately 20-25 new graduate students to the Ed.D. 
program. Based upon the cohort model in which students move lockstep through a carefully 
prescribed curriculum, the program admits a select group of talented leaders drawn from diverse 
organizational settings, including aspiring elementary and secondary public and private school 
administrators, postsecondary education administrators, government staff with education-related 
responsibilities, and private sector managers working in the field of education. Students enroll 
in six credit hours per semester for 3 years and undertake courses in leadership, organizational 
theory and behavior, social and political context of education, and research methodology with 
leading faculty members and distinguished education practitioners. Students complete the final 
capstone project in conjunction with coursework during their final year in the Ed.D. program. 

As part of the application process, prospective students submit an application, three letters 
of recommendation, academic transcripts from all postsecondary institutions, a resume, and a 
personal statement. Peabody College encourages all prospective students to schedule a campus 
visit to meet with program faculty, observe a class, and gain a better sense of the intellectual rigors 
and professional demands of the Ed.D. program. Successful applicants to Vanderbilt University’s 
Ed.D. program have master’s degrees, strong academic backgrounds, solid GRE scores, and 
demonstrated leadership capabilities at the elementary, secondary, or postsecondary level. It is 
assumed that students will remain employed in an education-related position throughout their 
tenure in the program and though the vast majority of Ed.D. students pay full tuition, the college 
offers a limited number of scholarships and merit aid for the most promising applicants. 

Once admitted, the department makes a concerted effort to support incoming Ed.D. students 
as they transition to graduate school. In May 2009, Peabody College launched a new orientation 
program for Ed.D. students and the revised program introduces incoming participants to other 
members of the cohort, departmental faculty, current students, and program alumni. A welcome 
packet consolidates all program-related information for students, provides an overview of aca- 
demic and student services, highlights key dates and administrative deadlines, and includes a 
directory of all enrolled Ed.D. students with their current positions and institutional affiliations. 
In short, the Ed.D. orientation program introduces new students to the intellectual life of the 
department and acquaints incoming students with the mission, key objectives, and expectations 
of the program. 

In conjunction with the revised orientation program, the department also introduced in 2009 
the Ed.D. Peer and Alumni Mentoring Networks. The department recognizes that the Ed.D. 
experience extends well beyond the classroom, and sustained interactions with fellow students 
and alumni constitute one of the greatest resources of the program. Therefore, the department 
developed new mentoring programs as a way to tap into those resources and to facilitate network- 
ing opportunities within and across cohorts. The Peer Mentoring Network assigns all incoming 
participants a student mentor based on career interests and affiliations and through department- 
sponsored social events, mentoring teams meet on an informal basis to share academic experiences 
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and professional challenges and build new relationships. On a practical level, the Peer Mentoring 
Network provides incoming students a friendly and supportive resource that can address ques- 
tions and concerns related to coursework, qualifying examinations, the capstone project, and 
the work-school-life balance. As the department continues to develop and refine its mentoring 
programs, it plans to invite program graduates to participate as alumni mentors. The mission of 
the Ed.D. program is perhaps best advanced by the cadre of talented alumni who work in a variety 
of education leadership positions, and the alumni mentoring program seeks to identify graduates 
who will provide counsel and assist current students in their professional development. 


SUSTAINING A COMMUNITY OF PRACTITIONER-SCHOLARS: 
ACADEMIC AND STUDENT SERVICES 


Unique administrative and logistical challenges attend the graduate school experience for working 
professionals in academic residence exclusively during weekends. To alleviate some of these ad- 
ministrative burdens, the program provides automatic course registration for all cohort members, 
facilitates the university activity/recreation fee waiver process for all nonresidential students, ne- 
gotiates special housing rates when classes are in session, and releases weekend course meeting 
dates a year in advance. The department enhances communication between and among students, 
faculty, alumni, and staff, through the Ed.D. Newsletter. Published three times per year to coincide 
with the start of the autumn, spring, and summer semesters, the newsletter provides information 
related to the Ed.D. program and to Vanderbilt University more generally. Further, through feature 
profiles of current students and alumni, the Ed.D. Newsletter provides another way for members 
of the Ed.D. community to learn about one another and stay connected. 

Ed.D. students also stay connected to the university and the program through active use of the 
Peabody Library. As the lifeblood of any academic institution, the library at Vanderbilt University 
serves an important function in sustaining the intellectual life of the Ed.D. program. The Peabody 
Library has created services exclusively for nonresidential Ed.D. students, which include the free 
overnight shipping of any book in the library collection, expedited shipping of volumes requested 
through Interlibrary Loan, and the conversion of any print article into electronic format. 


EXPANDING A COMMUNITY OF PRACTITIONER-SCHOLARS: ADJUNCT 
INSTRUCTORS AND THE Ed.D. NATIONAL ADVISORY BOARD 


The teaching demands of the Ed.D. program are largely met by the department’s tenured and 
tenure-track faculty members. But the scholar-practitioner model on which the program is based 
has compelled the program to integrate the scholarly expertise and professional insights of leading 
education practitioners. As key members of the Ed.D. faculty, adjunct instructors invigorate the 
practical nature of the program by exposing students to the problems of practice in contemporary 
education settings. For example, the program’s K-12 Education Law course is taught by a leading 
attorney with expertise in special education, religious issues, sexual harassment, teacher tenure, 
student discipline, and student rights. Similarly, the executive director of a leading state higher 
education commission teaches the College and University Finance course and brings expertise in 
state-level oversight, budgeting, and strategic planning. Our adjunct instructors help us to broaden 
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our course offerings at the same time that they reinforce the basic mission of the Ed.D. program 
by demonstrating how theory and practice are deeply intertwined. 

The Department further plans to develop an Ed.D. National Advisory Board composed of 
scholars and practitioners drawn from elementary, secondary, and postsecondary levels. Ed.D. 
Advisory Board members will be actively involved in the life of the program, and will serve 
as informal advisors to the program’s leadership and faculty. The board will also benefit our 
students. As national leaders in the educational community, board members will broaden the 
reach and professional networking opportunities for all Ed.D. students. 

By transforming its Ed.D. degree, Peabody College has created a truly distinctive doctoral 
program in education that serves a clear and distinct purpose: the training and preparation of the 
country’s next generation of education leaders. And as the department moves forward, it will 
continue to revisit the best ways to build, sustain, and expand an intellectual and professional 
community of education practitioner-scholars. 
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From Curricular Alignment to the Culminating Project: The 
Peabody College Ed.D. Capstone 


Claire Smrekar and Kristin McGraner 
Peabody College of Vanderbilt University 


The Department of Leadership, Policy, and Organizations (LPO) at Peabody College, Vanderbilt Uni- 
versity, recently replaced the conventional Ed.D. dissertation with a team-produced, client-consultant 
oriented, culminating report. This article describes the purpose and principles associated with the 
“capstone” project and describes the “fit” between the curriculum and the capstone experience. The 
chapter also describes the “lessons learned” from the initial cohort of doctoral students and clients 
who collaborated in the pilot year of capstone projects. 


INTRODUCTION 


In 2006, the Department of Leadership, Policy, and Organizations (LPO) at Peabody College, 
Vanderbilt University, launched a new doctoral project that replaced the conventional Ed.D. 
dissertation with a team-produced, client-consultant oriented, culminating report. This article 
describes the purpose and principles associated with the “capstone” project and specifies three 
courses that underscore the intentional “fit” between the curriculum and the capstone experience. 
The article also highlights elements that were modified after the initial series of capstone reports 
were completed. These changes reflect the “lessons learned” from the initial cohort of doctoral 
students and clients who collaborated in the pilot year of capstone projects. 


Rationale & Purpose 


The department’s decision to replace the single-authored, conventional five-chapter dissertation 
with a team-produced, client-centered capstone project reflects a compelling need to sharply 
differentiate the Ed.D. degree from the Ph.D. degree (Guthrie, 2009/this issue) and to connect 
more directly the culminating product of the practice-oriented Ed.D. to the expectations and 
demands confronting school leaders in the 21st century. In brief, this focus involves training 
leaders positioned to understand and improve student performance in socially and politically 
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TABLE 1 
Expectations, Orientation, and Format of Ed.D. Capstone Project Compared to the Conventional 
Ph.D. Dissertation 
a 
Conventional Peabody Peabody College Ed.D 
Ph.D. Dissertation Capstone Project 





Derived from or intended to contribute to theoretical Derived from client interests and intended to address 
explanations or concentrated upon policy problem operational issues 
of substantial state, national, or institutional 
significance 

Requires approval of four-person faculty committee, | Requires approval of capstone director and client 
one of whom is drawn from outside department 


Intended as cumulative contribution to knowledge, Present day, problem-centered orientation with 
grounded in prior research and relevant literature analysis linked to relevant literature 

Vanderbilt University Graduate School authority Peabody College authority and function 
and function 

Five chapter or three publishable paper format Management consultant report format 

Academic or scholarly in orientation and format Practically oriented and client centered 

Analytically rigorous Analytically rigorous and reader friendly 

Intended as a component of Professional Portfolio Intended as a component of Professional Portfolio 
and helpful to career advancement and helpful to career advancement 





complex organizations, enhance accountability and data collection systems, and produce effective 
and efficient governance structures. Table 1 highlights the differences between the conventional 
Peabody College Ph.D. dissertation and the Ed.D. capstone project. 

The capstone idea reflects a consensus view among faculty at Peabody College that the cul- 
minating analytical experience should prepare educational leaders to exemplify a skill set that 
includes deep knowledge and understanding of inquiry, organizational theory, resource deploy- 
ment, leadership studies, and the broad social context associated with problems of educational 
policy and practice. The department situated the capstone within a comprehensive, locked-in 
curriculum (Caboni, 2009/this issue) designed to provide the scaffolding for a problem-centered, 
practically oriented professional doctorate in educational leadership. The capstone provides the 
pivot point to advance students along the pathway from doctoral training to high performance 
management, professional practice, and leadership. In sum, the primary objectives of the cap- 
stone are to produce educational leaders who have informed, critical, and creative approaches to 
understanding and addressing complex educational problems. 


Key Principles and Components 


LPO has two capstone directors; one is a faculty member who specializes in K-12 leadership 
and policy, and the other is a faculty member who focuses on higher education leadership and 
policy. The directors coordinate all capstone project development and supervise capstone projects 
within the domain of specialization. Each year, LPO capstone directors engage in the following 
activities: (a) initiate contact with potential clients; (b) develop a specific capstone project in 
consultation with clients; (c) schedule capstone presentations to the third-year student cohort and 
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coordinate students’ match to specific teams; (d) facilitate capstone team-client contact; and (e) 
supervise capstone projects and co-instruct the capstone seminar. 


Client Development. Potential clients for the capstone projects are cultivated through pro- 
fessional contacts and previous work associations, faculty suggestions, and independent develop- 
ment activities undertaken by the capstone directors. Recently, a small number of clients contacted 
Peabody directly with interest in possible capstone collaborations. 

Clients may include school districts, state education departments, state education commissions 
and other state-level education agencies and governing bodies, education associations and orga- 
nizations representing specific types of schools (e.g. independent, military-sponsored, Catholic, 
rural), higher education systems and institutions, and international education/economic develop- 
ment organizations. 

Previous clients include school districts in Austin, TX; Montgomery County, MD; Nashville, 
TN: and Jefferson County, KY; the Southern Association of Independent Schools; the Tennessee 
Higher Education Commission; Tennessee Board of Education; Tennessee Board of Regents; 
Rhodes College; Middle Tennessee State University; Vanderbilt University; and the Tennessee 
Independent Colleges and Universities Association. Some clients are retained for capstone de- 
velopment and collaboration the following year; each year, however, a subset of new clients is 
selected for participation in the capstone. 


Client Selection & Project Adoption. Clients are selected each year based upon scope, 
rigor, and relevance of the project to the interests and program priorities in LPO. A “Request 
for Assistance” (RFA) drafted by LPO guides potential clients and prospective projects. The 
RFA specifies a range of projects, performance expectations, consultation and communication 
processes, and final report guidelines. In consultation with the LPO capstone directors, a final 
“proposal” is drafted by the client and screened by directors before selections are finalized. All 
capstone projects are presented to the Ed.D. third-year cohort members by clients. Presentations 
take place on the Peabody campus in the summer (June) before the beginning of the final year of 
Ed.D. students’ program. 

Each year, approximately six to seven (total) projects are selected for the K-12 and higher 
education capstone projects, although the total number of proposals presented each year may 
exceed the total number actually adopted by Ed.D. student participants. The final number of 
capstones adopted reflects the size of each Ed.D. cohort, typically 18-21 students. Each capstone 
team is comprised of 2 or 3 students, each of whom rank-order (one through three) their preferred 
projects. Capstone directors assign students to teams based upon students’ personal preferences, 
with the goal and expectation that most students will be matched with either their first or second 
choice. This goal has been met each year. 


The Capstone Seminar. Students enroll in a two-semester, three-credit capstone seminar 
in the final year of the (3-year) doctoral program. The seminar is designed to provide a formal 
structure and set of integrated, benchmark assignments that culminate in the formal presentation 
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and publication of the final capstone report. The capstone seminar is comprised of the following 
elements: 


Regular Communication. A minimum of two meetings between capstone directors and 
project teams are scheduled for convenience in conjunction with students’ regular weekend 
courses throughout the fall semester. Some meetings may be arranged as conference calls. Cap- 
stone directors encourage project team members to remain in regular contact with one another 
(through group meetings, E-mail, and telephone) throughout the project’s duration. 


Scope of Work (SOW) Memo. The Scope of Work (SOW) is an integral part of the early 
stages of the capstone. Students are required to submit the SOW within a month of the June client 
presentation meeting and team formation. The four- to five-page SOW involves: (a) defining 
the scope of the project, including key questions and contexts (temporal, comparative, local, 
statewide, national); (b) specifying the analytical focus and data collection strategy: reports, 
memos, records, research literature, data files, surveys, interviews; (c) developing a timeline and 
task completion schedule; and (d) assigning team members to specific tasks. 

Upon approval of the SOW memo by the relevant capstone director, students begin to imple- 
ment their project plan. This stage involves reviewing relevant documents, reports, and research 
literature; developing instruments (e.g., survey, interview protocol); identifying key stakeholders 
and relevant research subjects; and beginning data collection. An application to the Institutional 
Review Board (IRB) in compliance with human subjects protection policies is made upon ap- 
proval of the SOW and completion of all relevant data collection instruments. 


Student Effort Reports. Individual reports are collected at three intervals from each mem- 
ber of all capstone project teams to gauge level of effort and contribution among individual mem- 
bers. Reports are collected midway through the summer, fall, and spring academic semesters. 
Capstone directors and the LPO department chair meet with any student whose level of contri- 
bution, effort, and quality of performance are identified by other project team members as below 
expectations and insufficient. Corrective action is imposed if these issues arise. 


Status Reports. A status report that describes the instruments developed and data collected 
(report I) and initial analyses (report II) is collected at two points in the fall academic semester 
and is designed to pace students along a path that will culminate in completion of a draft report 
by late February. 


Interim Reports. LPO capstone directors convene a second capstone project meeting (full 
cohort) in early January. Students present interim reports (types of data collected and analyzed, ini- 
tial findings, discussion of the findings and linkages to extant research, initial recommendations), 
identify the work to be completed, and share plans for report writing and presentation. Student 
presentations (PowerPoint) and capstone directors’ feedback and discussion are scheduled in 
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60-minute intervals throughout the Friday evening and all-day Saturday event. Draft reports are 
due 6 weeks later. 


Draft Reports. A draft report is due in late February. Draft reports include recommendations 
to the client that reflect the analyses and findings in the report. Students are reminded that it is 
customary in a consulting project that clients are free to dispute findings and reject recommenda- 
tions. Capstone directors review, provide feedback, and return draft reports to student teams by 
early March. Completed (not necessarily final) reports are presented publicly 6 weeks after the 
draft reports are returned to students. 


Capstone Reports and Presentations. Students present a completed capstone project to 
LPO doctoral students and faculty in mid-April. Prior to or immediately following this “public 
hearing,” students provide a “private briefing” to the client at a time and location of mutual 
convenience. This briefing may be scheduled as a conference call or an in-person presentation. 
Following the public presentation and based upon faculty input and reaction, capstone project 
reports are edited to incorporate suggested changes. Final reports are due to the LPO capstone 
directors by the first of May. 


Final Capstone Reports. Each final capstone report is approximately 75 pages (single- 
spaced, excluding appendices). Reports include multiple sections, depending upon project par- 
ticulars, including: (a) definition of the problem or issue; (b) contextual analysis of the problem; 
(c) findings (e.g., financial, operational, statistical, evaluative, qualitative, demographic); (d) dis- 
cussion of key findings; (¢) recommendations; (f) implementation strategy; (g) conclusions; (h) 
appendices; and (i) references. Students are encouraged to consider double-column or other desk- 
top publishing formats that reflect a professional (annual) report style, with tables, graphs, and 
quotes set off in text with color printing. 


\llustrative Capstone Projects 


Although capstone projects vary by focus area (K-12 or higher education), geographical location 
(Tennessee or another state), institution (school, system/district, agency, association), and scope 
(case study, systemic review, program evaluation, environmental scan/program proposal), all share 
4 set of common characteristics related to rigorous analysis in a realistic operational setting. The 
capstone represents an opportunity for Ed.D. students to apply analytical capacities, professional 
knowledge, contextual understandings, and teamwork skills acquired and accumulated throughout 
the Ed.D. program to a focused project undertaken for a real-world client. Below is an illustrative 
Request for Assistance (RFA) and Executive Summary description (prepared by the capstone 
team); collectively these documents capture both the common and distinctive qualities of the 
Peabody capstone. (All capstone projects are available on e-archives.) 
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EXHIBIT | 
Metropolitan Nashville Public Schools (MNPS) 


Request for Assistance (RFA). A follow-Up Study on Data-Driven Decision-Making 
Practices. During the 2006-07 school year, Metropolitan Nashville Public Schools (MNPS) 
collaborated with a team of Ed.D. students from Vanderbilt University’s Peabody College to 
evaluate the data-driven decision-making (DDDM) practices of district principals. Over the 
course of the school year, this team conducted surveys and interviews of MNPS educators to 
learn more about principals’ capacity for DDDM practice and their ability to transfer those 
skills to their teachers. The team produced a series of useful findings and devised a number of 
recommendations to facilitate improvement of DDDM in MNPS. Capstone team members will 
analyze this earlier report as a basis for proceeding with the follow-up 2007-08 capstone in 
MNPS. 

This follow-up capstone project involves in-depth, qualitative analyses of teachers’ DDDM 
practices. The first capstone project focused primarily on principal data use and capacity but did 
not provide the depth of information to adequately understand how and why teachers use data in 
their professional practice. This RFA is designed to build upon the earlier analyses by examining 
DDDM at the classroom level, where teacher DDDM meets student learning. 

As part of this capstone project, a team of Ed.D. students will conduct a series of qualitative 
analyses (e.g., teacher interviews, classroom observations, document analyses) to explore teacher 
knowledge, attitudes, and behavior related to DDDM. 


Contact: 
Dr. Paul Changas, MNPS 
Office of Assessment and Evaluation 


EXHIBIT II 


Examining the Roles of Site Coordinators and School Counselors in the 
Development and Implementation of Program Initiatives (Sean Chapman, Kate 
Donnell & Kristin McGraner, 2008) 


This report examines the role of GEAR UP site coordinators and school counselors in the devel- 
opment and implementation of GEAR UP Tennessee. The GEAR UP Tennessee program is an 
ambitious effort that offers a myriad of interventions to support academic preparation and college 
access in rural communities across the state. Though supported by a network of local and state 
partners, the program gives the nine participating districts discretion in the design and imple- 
mentation of initiatives at the local level. Site coordinators are the primary agents charged with 
the responsibility of district-level implementation. Within the school context, school counselors 
are the individuals with the organizational proximity necessary for meaningful interactions with 
students concerning educational advancement. Though GEAR UP Tennessee has collected data 
relative to the program’s effects on schools, teachers, and parents, the work of site coordina- 
tors and school counselors has been largely overlooked. Therefore, in response to a request for 
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assistance from the Tennessee Higher Education Commission (THEC), the team developed the 
following research questions: 


1. How does the program structure and district context shape site coordinators’ implemen- 
tation of GEAR UP? 
2. What factors affect school counselors’ implementation of GEAR UP initiatives? 


The team conducted 63 semistructured interviews with GEAR UP site coordinators, district 
leaders, school principals, school counselors, and THEC officials. Interviews were designed 
to gather information on the district’s performance in preparing students for postsecondary 
education; the respondent’s knowledge of and role within GEAR UP; district and school supports 
and barriers affecting implementation, which include issues around individual and institutional 
capacity and will; the coherence of program messages and the sense-making in which respondents 
engaged to make decisions about their participation in the program and its implementation; and 
respondents’ perceptions of program effects. The data reveal that: 


e Participating school districts were pressured by, and as a result largely focused on, No 
Child Left Behind (NCLB) compliance. The presence of NCLB largely detracted from the 
district’s ability to fully embrace GEAR UP and integrate it into their district improvement 
plan. 

e District and school leaders possessed little knowledge about GEAR UP, its intended 
implementation, and the appropriate role structure of site coordinators, district personnel, 
and school personnel. 

© Most site coordinators did not perceive GEAR UP as a potential lever for systemic change. 
Site coordinators and school counselors received little substantive support from state and 
local leadership relative to implementation of GEAR UP college access interventions. The 
content of site coordinators’ work focused predominantly on activity planning, resource 
distribution, and grant compliance. 

e The community culture, specifically the “welfare state of mind.’ was perceived by all 
respondents as a barrier to advancing students’ educational attainment. 

e Training and professional development activities have been provided for site coordinators 
with a primary focus on grant compliance and reporting mechanisms. Site coordinators 
reported few opportunities to deepen their knowledge of how to increase students’ aca- 
demic preparation and college access, which has significant effects on implementation 
outcomes, program sustainability, and systemic change. 

e Training for district and school personnel has been lacking and, in many cases, nonexistent. 

© The work of school counselors is influenced by the lack of a coherent counseling cur- 
riculum, time constraints, and role ambiguity. Consequently, counselors provide sporadic 
support and leadership in GEAR UP implementation. 


As a result of these findings, the team offers the following recommendations to ensure full 
program implementation and the attainment of program goals: 


e Improve the visibility and effectiveness of site coordinators by developing communication 
networks among coordinators and school and district personnel; creating comprehensive 
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training manuals for coordinators; and implementing a series of trainings that address the 
factors influencing students’ academic preparation and college access, as well as program 
implementation strategies supportive of GEAR UP goals. 

¢ Educate and involve district and school leadership by developing aGEAR UP TN Leaders 
Guide in tandem with GEAR UP TN leadership trainings. Trainings will create the forum 
to collaboratively plan with site coordinators to align GEAR UP TN interventions to both 
the district and school improvement plans. 

e Educate and involve the school counselor by developing a GEAR UP TN School Coun- 
selors Guide in tandem with GEAR UP TN counseling trainings. Trainings will help 
counselors implement college preparation curricula, improve collaborative planning with 
site coordinators, and develop communication networks among counselors. 

¢ Collectively develop a comprehensive sustainability plan that determines how to effectively 
disseminate data; galvanize support of the school counselor as well as district and school 
leadership within GEAR UP TN; and affect state-level policy change to enhance the core 
goals of GEAR UP TN. 


Instructional Strategies Aligned With the Capstone 


This section provides a descriptive analysis of the alignment between Ed.D. courses and the 
scope, requirements, and expectations of the capstone. The courses described here are exemplars 
but reflect a common set of objectives related to the capstone and the Ed.D. curriculum. Three 
illustrative courses are noted: a K-12 specialty course, a core course required for both K-12 and 
higher education students, and a research methods course required of all Ed.D. students. 

The Social Context of Education is designed to “explore contemporary social, philosophical, 
and political dimensions of education and their relationship to leadership, including issues re- 
lated to democracy and diversity, equity and school organization, and the ecology of schooling” 
(Smrekar, 2008, p. 1). In addition to students examining the scholarly literature in this area, they 
explore the opportunities and limitations of educational leadership and organizations for address- 
ing major social, political, and philosophical dilemmas (Smrekar, 2008). This course objective 
reflects the integration of theoretical knowledge and practical application, and distinguishes the 
new Ed.D. program from previous models. 

The Social Context course includes three major projects requiring students to (a) develop a civic 
engagement program proposal with an analytical review of the literature and existing program 
exemplars; (b) author a paper for a mock panel discussion at a national education conference on 
the ability of schools to eradicate social inequities; and (c) draft a report that addresses student 
assignment/admissions policies from the perspective of a task force on achieving diversity in 
education. Each of these projects requires students to examine the relevant literature and utilize 
research to inform coherent, thoughtful, and well-developed papers designed to address authentic 
and current problems of policy and practice. Situating students in leadership roles as members of 
education agencies demands that they tend to the practical implications of solutions they develop. 
Moreover, the course prepares students to engage in professional and grounded conversations in 
complex and high-stakes environments, which is a key feature of the capstone project. 

The Teachers and Teaching course is designed to bolster students’ understanding of instruction 
and the type of instructional leadership struggling schools necessitate. The coursework is designed 
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to provide grounding in what educational leaders should know about research on teaching, teacher 
learning, what administrators need to know to be effective instructional leaders, and the challenges 
of taking an instructional reform to scale (Smith, 2008). Further, students are asked to think 
critically about the following questions that ground the course: 


e What is high-quality teaching? How do you know it when you see it? 

e How can we best support both new and experienced teachers in developing high-quality 
instructional practices? What types of organizational and policy supports are needed to 
make teacher professional development more effective? 

© What is instructional leadership? What type of content knowledge do leaders need to 
effectively lead instructional change? How do instructional leaders make use of that 
knowledge in their everyday practice? 

© What are the challenges for instructional leaders in taking an instructional reform to scale? 


The readings in the course challenge students to address these questions using extant re- 
search in the areas of leadership, teaching, and learning. Students demonstrate understandings 
through class discussions, online Blackboard reflection postings, and the completion of three 
projects. The first project requires students to analyze and critique existing teacher evaluation 
instruments with a focus on the types of knowledge and leadership strategies necessary to use 
these tools effectively. Second, students are asked to interview a state or district leader charged 
with designing and implementing professional development in their respective settings. Then, 
using interview data and artifacts, students evaluate the professional development program and 
construct recommendations for improvement. 

Finally, the third project requires students to critique the construct of leadership content 
knowledge (Stein & Nelson, 2004) and interview educational leaders to discern the depth of their 
knowledge, as well as develop a professional development program for school leaders with the 
intent to strengthen their content knowledge and capacity for leading learning across the content 
areas. 

The Qualitative Research Methodology course is designed to familiarize students with the 
fundamentals of conducting qualitative research. Through course readings, class discussions, and 
a team-based, real-world research project, students gain the knowledge and skills required to 
design and conduct qualitative research. Furthermore, students engage in critiques of qualitative 
studies with specific attention to the ways in which the authors frame research questions, select 
methodologies, recognize limitations, and ensure reliability and validity. Thus, students are not 
only prepared to conduct high-quality research, but they are trained to act as wise and critical 
consumers of research and to apply research to their professional practices in strategic and logical 
ways. 

Students are assigned to project teams of two or three members each. Each team is required to 
design and conduct a small-scale research project for a real client. For example, in 2006 the THEC 
requested assistance from the teams in examining students’ college awareness and understandings 
of financial aid, particularly students’ knowledge of the state merit- and need-based scholarship 
program. The teams comprised of K-12 Ed.D. students focused on a diverse set of high schools 
across Nashville and interviewed high school seniors, teachers, and school counselors. The higher 
education teams interviewed college freshman and staff members in the offices of financial aid 
and academic support at 4-year institutions and community colleges across the state of Tennessee. 
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This project integrated knowledge gained across the curriculum (including Social Context and 
Teachers and Teaching), formed the basis for learning about research methods, and encouraged 
students to delve more deeply into important issues confronting K-12 and higher education. 

In this course, students develop research questions, interview protocols, and analysis plans in 
consultation with other members of the class, the professor, and the client. The culmination of 
the client-centered, real-world analytical experience in this course is a report tailored specifically 
to an intended audience. The report structure mirrors the capstone project structure, which 
includes an introduction and framing of the problem, project questions, conceptual underpinnings, 
methods, findings, discussion, and recommendations. Students are pushed to develop actionable 
recommendations that stem from the findings. The team-based, client-centered report provides 
students with an initial guiding experience in preparation for the Ed.D. capstone project. 


Curricular Coherence and the Capstone 


The Social Context of Education, Teachers and Teaching, and Qualitative Research Methodology 
courses exemplify the direction of the new education doctorate and align with the Peabody 
Ed.D. capstone project. Fundamentally, these courses are “professionally anchored” (Murphy 
& Vriesenga, 2005), meaning that they are grounded in authentic policy and practice dilemmas 
that educational leaders encounter in their professional contexts. The courses’ expectations and 
products mirror the capstone project in several critical ways. First, students are positioned as 
educational leaders and consumers of data and research throughout their coursework. They learn 
the types of questions to ask and the types of data they need to collect to best inform their leadership 
approach. For example, Teachers and Teaching asks students to gather artifacts and interview data 
while also using the extant literature on professional development to develop recommendations 
for how to better design learning opportunities for teachers. Therefore, students are acting as 
evaluators and wise consumers of research while exercising their academic sensibilities within 
environments typified by ill-defined problems of policy and practice. Training candidates in this 
way guards against “the danger that we achieve rigorous preparation neither for practice nor for 
research” (Shulman, Golde, Bueschel, & Gorabedian, 2006, p. 26). 

Parallel to the core features of the capstone, the Ed.D. curriculum requires that students 
integrate their knowledge across disciplines, develop sound research-based frameworks for un- 
derstanding and addressing problems of practice, and develop actionable recommendations for 
reform. The Social Context of Education course, for example, asks students to analyze post- 
desegregation student assignment plans and provide not only theoretical rationales and under- 
pinnings and, more importantly, requires that students construct a strategy and policy initiative 
to address the dilemma. Furthermore, the Qualitative Research Methodology course requires 
students to synthesize their learning in the social context of education, education governance, re- 
source allocation and deployment, and leadership theory and behavior to study the issues around 
need- and merit-based scholarship programs. More specifically, students in this case examine 
how schools are preparing students for postsecondary education and the interplay among public 
policy, economic and social development, and educational leadership. In so doing, students syn- 
thesize knowledge across disciplines as they design policy and reflect on its implementation and 
outcomes. 
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LESSONS LEARNED 


After three cohorts, careful analysis, reflection, and feedback, the Peabody College capstone 
project has been refined in response to a set of issues highlighted by an array of stakeholders, 
including students, faculty, capstone directors, and clients. The issues identified include: (a) 
client vetting; (b) client-consultant relations; (c) client anonymity; (d) role of faculty advisors; 
(e) timing of capstone client presentations; and (f) consequences of “failed” capstones. These 
six elements serve productively as a template for “best practices” and strong recommendations 
for those institutions considering the capstone project as the culminating element of the Ed.D. 
program. 


Client Vetting 


One of the most important elements of the capstone project is clarity and transparency, with 
specific reference to the expectations and motivations of clients. The first year of capstone projects 
provided an illustrative example of problems that arise when clients position the institution or 
system to protect from review and analysis relevant, but particularly problematic, aspects of 
the organization, leadership structure, program, or personnel. Clients are now asked a series of 
questions designed to delineate the proposed scope of work envisioned in the capstone (rather than 
assume congruence and shared understanding based upon informal discussions), the rationale for 
engaging in the collaborative project with Peabody College, and the anticipated use, distribution, 
or application of the capstone report analyses within the organization. 


Client-Consultant Relationships 


Peabody has learned from the past three student cohorts the importance of clarifying for Ed.D. 
students and clients their respective roles, with specific reference to reporting results and outlin- 
ing recommendations in the final report. In the first year, one student team acknowledged that 
they withheld some findings and recommendations in anticipation that these results would create 
uncomfortable conditions for some members of the client organization. As is customary in a 
consulting-client relationship, clients are always free to dispute findings and reject recommenda- 
tions. We now make this clear to all participants. 

The need to specify the parameters of “access to client data is clear. This issue is best addressed 
formally through a “Memo of Understanding” between the capstone team and the client. This 
step clearly identifies the capstone teams’ access to types of data, personnel, confidential, and 
archival information relevant to the Capstone Scope of Work (SOW). 


Client Anonymity 


As part of the capstone seminar in the final spring semester, students provide a “private briefing” 
to the client at a time and location of mutual convenience following the completion of the report. 
Far in advance of this meeting, at the start of the capstone project, doctoral students and clients 
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discuss the potential public use, distribution, and publication of the report. These decisions rest 
with the client and are reached at their discretion; some clients prefer or are ambivalent regarding 
the use of actual school and organization names; others insist upon anonymity and the use of 
pseudonyms. Students are instructed to discuss issues of anonymity and confidentiality at the 
initial meeting with clients. 


Role of Faculty Advisors 


Peabody capstone directors provide general guidance and feedback on all projects within the 
K-12 and higher education program areas. In addition, the directors regularly assign a “faculty 
advisor” in the LPO Department to capstone projects that fall outside the directors’ specific areas 
of expertise. The faculty advisors provide critical feedback on survey design, relevant literatures, 
and complex conceptual issues throughout the capstone process. This additional layer of faculty 
support and supervision ensures analytical depth and rigor across all capstone projects. The role 
and function of faculty advisors requires selection Strategies that intentionally match specific 
capstone topic areas with the domains of expertise arrayed across the faculty in the department. 


Timing of Capstone Presentations 


The length of time designated for the capstone project, from beginning to end, was lengthened 
after the first year to more appropriately reflect the time required to complete high quality, field- 
based, client-centered projects. Client presentations to the Ed.D. cohort occur in the first week 
of June (rather than September); final presentations are made 10 months later, in mid-April, and 
final project submissions are due the first week of May. The entire process for the student spans 
11 months. 


Consequences of “Failed” Capstone Projects 


The assurance of a highly ranked capstone project (rather than a conventional dissertation) is not 
a guarantee of graduation at the end of the third and final year of the Ed.D. program. Peabody 
College recognizes the need to establish clear guidelines for completed capstone projects that 
are consonant with the expectations of rigor and scope associated with all doctoral-level work, 
including conventional dissertations and capstone projects. Capstone directors assume primary 
responsibility for assessing the quality and acceptability of all projects, in consultation with each 
client. If a capstone project is deemed unsatisfactory by the director or client following the April 
presentation and subsequent recommendations for improvement, team members will receive an 
“Incomplete” grade for the three-credit capstone seminar. These students will have not fulfilled 
the requirements necessary to graduate in May. As a result, graduation will be delayed until the 
capstone project is accepted, the grade in the seminar is changed to a “Complete,” and all doctoral 
requirements are fulfilled (presumably at the conclusion of the summer semester in August or 
thereafter). 
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This article describes the process by which the faculty of the Department of Leadership, Policy and 
Organizations at Vanderbilt University’s Peabody College conceived and implemented a new doctoral 
program designed for senior education practitioners. The authors describe the organizational context in 
which decisions were made. Next, they explain the theoretical and philosophical underpinnings of the 
revised degree program. Then, specifics of the program curriculum and delivery are detailed. Finally, 
the article explores how the reconceptualized program is differentiated from the traditional research 
doctorate. At Vanderbilt, the Ed.D. is designed for mid-career professionals with an orientation to 
practice who seek a terminal degree and career advancement within their organizations. It is oriented 
toward solving problems of practice and takes as its goal placement of graduates into senior leadership 
positions within educational organizations. 


INTRODUCTION 


This article describes the process by which the faculty of the Department of Leadership, Policy 
and Organizations at Vanderbilt University’s Peabody College conceived and implemented a 
new doctoral program designed for senior education practitioners. The authors describe the 
organizational context in which decisions were made. Next, they explain the theoretical and 
philosophical underpinnings of the revised degree program. Then, specifics of the program 
curriculum and delivery are detailed. Finally, the article explores how the reconceptualized 
program is differentiated from the traditional research doctorate. 


ORGANIZATIONAL CONTEXT 
Vanderbilt University 


Vanderbilt University is a private university in Nashville, Tennessee, founded in 1873. It comprises 
four undergraduate and six graduate schools, enrolling approximately 11,800 students from all 
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50 USS. states and over 90 foreign countries. Fifty-five percent of students come from outside the 
southeastern United States and 8% of the student body comes from outside the United States. 
In its 2009 ranking of universities, U.S. News and World Report placed Vanderbilt 18th among 
national universities. 


Peabody College 


George Peabody College for Teachers, recognized for more than a century as one of the foremost 
independent colleges of teacher education, merged with Vanderbilt University in the summer of 
1979 to become Vanderbilt University’s Peabody College of Education and Human Development. 
Since that time, Peabody College has retained its heritage while achieving new stature as a national 
leader among schools of education. 

The college, embodied in its faculty and students, constitutes a vibrant intellectual community 
dealing with pressing questions and expanding knowledge about education, including special 
education; psychology, especially focused on families and children; the development of indi- 
viduals, organizations, and communities; and education administration, leadership, and policy. 
Peabody enrolls 684 graduate and professional students and 1,202 undergraduates. More than 
10,000 Peabody alumni are practicing teachers, more than 175 are school superintendents, and 
more than 30 are college or university presidents. 


Department of Leadership, Policy, and Organizations 


The department of Leadership, Policy and Organizations (LPO) takes as its mission understanding 
and enhancing the social and organizational contexts in which learning occurs. The department’s 
research efforts and Ph.D. preparation programs fall within the umbrella of “understanding,” and 
the efforts related to M.Ed., Ed.D., and professional development programs are included within 
the “enhancing” portion of the department’s mission. The department enrolls 44 Ph.D. students 
and 215 professional students, of whom 56 participate in the Ed.D. program. 


Graduate and Professional (Ed.D.) Reform 


In response to a university-wide examination of Ph.D. programs and recommendations included 
in the final report of the Graduate Education Task Force, during academic year 2002-03, LPO 
substantially revised its Ph.D. programs. The curricular focus of the department’s Ph.D. programs 
shifted exclusively to the preparation of future Research I faculty members and senior-level policy 
analysts for government and select nongovernmental organizations. 

This curricular revision effectively removed Ph.D. students from courses once populated by a 
mix of individuals pursuing M.Ed., Ed.D., and Ph.D. degrees. The redesign also had the added 
effect of rendering impossible student transfers from the department’s Ed.D. program into the 
Ph.D. program. Though not a tremendous problem, frequently full-time enrollees in the Ed.D. 
program (who took many of the same courses as Ph.D. students) attempted to transfer into the 


RE-ENVISIONING THE PROFESSIONAL DOCTORATE FOR EDUCATIONAL LEADERSHIP 63 


Ph.D. program. L. D. Brown (1990) suggests that the status differential between the two degrees 
accounts for the preference among students for the Ph.D. 

After revising the Ph.D. program, LPO faculty had an opportunity to rethink the department’s 
doctoral programs for practitioners. During the summer of 2003, an initial planning meeting was 
held where 24 LPO faculty participants appraised practitioner or professional degree program 
alternatives along with potential college professional development and outreach activities. These 
planning activities extended through academic year 2003-04. The Peabody College Faculty 
Council granted final approval of a redesigned program in spring 2004 for the entering class of 
Ed.D. students to begin study the following summer. 


THEORETICAL AND PHILOSOPHICAL UNDERPINNINGS 


Common descriptions of the differences between Ed.D. and Ph.D. degrees, and arguments ad- 
vocating the need for both, typically include the rationale that the Ed.D. prepares K-12 and 
college and university administrators for practice careers informed by theory, whereas the Ph.D. 
concentrates on training faculty and researchers (Watts, 2000; Weidman, Twale, & Stein, 2001). 
However, the difference between the two degrees has long been problematic, with little dif- 
ferentiation at most institutions as far back as the 1930s (Clifford, & Guthrie, 1988: Cremin, 
1978). 

At many universities where both the Ed.D. and Ph.D. are offered, there is a great deal of 
overlap between both degrees, even in many cases with identical research methods courses taken 
by students across programs (Osguthorpe & Wong, 1993; Richardson & Walsh, 1978; Shulman, 
Golde, Bueschel, & Gorabedian, 2006). As described above, Vanderbilt also faced this challenge. 
Without real clarification in terms of curriculum, career path, and outcome, the Ed.D. will 
continue to face challenges and lack wide acceptance as the primary degree for practice-oriented 
individuals seeking a terminal degree in education. 

During the process of revising the program, faculty discussions centered on the need to 
create a “lighthouse model” that might set an example for other schools of education. Given 
the isomorphic nature of colleges of education (DiMaggio & Powell, 1983; Toma, 2002), those 
institutions attempting to increase their national regard might emulate a Vanderbilt program. A 
well-designed and well-implemented program would have the potential to shape the landscape 
of professional education degree offerings nationally. 


PROGRAM DESCRIPTION AND SPECIFICATION 
Practice Focused Model 


Key to the reconceptualization of the Vanderbilt education doctorate was the primacy of problems 
of practice in the preparation of school and university leaders. Universities, especially research 
universities, tend to privilege the theoretical in instruction. Yet writing research papers, taking 
traditional statistics courses, and writing a dissertation have limited long-term utility for individ- 
uals leading education organizations (especially those who are active practitioners during their 
studies). 
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FIGURE 1 Ed.D. conceptual framework. 


Transferring learning from a classroom context to the “real world” is frequently difficult for 
students (Butterfield & Nelson, 1989). However, if instruction is delivered in such a way that 
students grapple with problems that emerge from the challenges of a school or university, solve 
those problems using the tools they would use in practice, and eventually produce final products 
with relevance to, and usefulness for, the site from which the problem came, learning is more 
likely to be obviously transferable (J. Brown, Collins, & Duguid, 1989). This requires faculty to 
teach in such a way that knowledge is both useable and adaptable. 

This provided us with a conceptual framework (Figure 1) to guide our process. The framework 
begins with students studying the relevant theoretical literature to develop a robust understanding 
of the knowledge base which informs administrative practice. They also receive training using 
data analysis techniques that can be used to answer questions related to challenges faced by 
real schools or colleges. Finally, they learn to contextualize and tailor solutions to the specific 
challenge to which, and institution in which, they are proposing solutions. 

As the LPO faculty worked to craft the specifics of the program, there were several major 
features that the faculty stipulated at the start. Each of these provides a major comparative 
advantage in the marketplace and helps to differentiate the program from the department’s 
previous offerings. 


Practitioner Oriented. The strong practitioner orientation helps to ensure that there is a 
clear distinction from the Ph.D. program, which is designed as a faculty preparation program. 
The goal was to create an equally rigorous program with a practice orientation that would not 
appeal to those interested in faculty positions; rather, it would attract students interested in 
pursuing (or currently occupying) senior administrative posts. Illustrative positions of those for 
whom the program is appropriate include: a 
e+ 
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¢ Current and aspiring elementary and secondary education administrators 

Current and aspiring elementary and secondary independent and private school adminis- 
trators 

Postsecondary education administrators 

Government officials with education-related responsibilities 

International educators, government officials, and agency senior staff 

Private sector managers with education-related responsibilities. 


Restricted Admission. The program was designed to ensure that only highly qualified 
applicants were admitted to study. As a result of limited capacity in the curriculum as designed, 
the maximal number of enrollees which could matriculate in a given year was limited to 24-30 
(12-15 each in the Educational Leadership and Policy and Higher Education Leadership and 
Policy programs). This guarantees that the program remains highly selective and that participants 
receive individual attention from core faculty across courses and during the final project. 


Low Participant Foregone Income. One of the challenges to recruiting mid-career admin- 
istrators to a doctoral program is the opportunity cost associated with stopping-out of the work- 
force to return to full-time study. To mitigate this, the curriculum was designed as a weekend-only 
curriculum where participants commute to campus six weekends per semester. This has the ad- 
ditional benefit of keeping enrolled students engaged in administrative practice for the duration 
of their course of study, which adds real-world complexity and practical focus to classroom 
discussions. 


Cohort Based. The program follows a strong cohort model with students taking all courses 
together with members of their cohort for the entirety of the program (24 credit hours are taken 
with the full cohort and the remaining 30 credit hours are taken with one’s specialization cohort). 
This creates a strong, professionally oriented learning community that lasts beyond completion 
of the degree. This model has the added advantage of providing a lockstep curriculum with no 
electives, ensuring that students who successfully complete all evaluations on time will graduate 
in 3 years. 


Problem-Oriented Curriculum. The curriculum is “professionally anchored” (Coghlan & 
Brannick, 2005; Murphy & Vriesenga, 2005), meaning that each course is situated among real- 
world challenges faced by practitioners during the course of their daily work. Though course titles 
may seem similar to courses offered in a typical administrator preparation program, pedagog- 
ically they are quite different. Coursework focuses on problem finding within organizations— 
specifically identifying the “correct” problems, avoiding misdiagnosis of core problems, and 
parsing those organizational dilemmas into something that can be addressed systematically. 


National Advisory Board. The reconceptualized Ed.D. program has a national board of 
advisors comprised of senior practitioners in both K-12 and higher education. Assembled with 
an eye toward infusing a strong practice orientation in all of the program coursework, the board 
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meets twice per year to provide guidance and evaluation’ of program efforts, provide input into 
the value and construction of the capstone projects, and add additional national visibility and 
credibility among the practice community. 


Regular Faculty Engagement. As we engaged in discussions about how we might most 
effectively deliver courses for working professionals and decided to move in the direction of a 
weekend-only program, the faculty remained steadfast that full-time core college faculty mem- 
bers would primarily teach the courses. Of the 26 courses in the program offered across both 
educational leadership and higher education leadership, only 3 are taught by instructors other 
than full-time Vanderbilt faculty members. This orientation ensures that we continue to immerse 
students in the theoretical literature without being pulled toward a more “nuts and bolts” type of 
coursework. 


Intellectual Core and the Curriculum 


In our discussions, we began with the skill sets that practitioners would need for success in 
the field and what a culminating project would look like at the end of the program. We agreed 
that the degrees should prepare students for the work they would eventually do as practitioners. 
We grappled with why we had our Ed.D. students engage in a traditional dissertation when the 
skills developed had little utility for the problems of practice confronted by administrators. We 
were careful not to revert to the traditional dissertation form and focused on a problem-based 
project grounded in the reality of organizational life. From this final capstone experience, we 
backward-mapped the curriculum to deliver the necessary skills in the curriculum. 

The revised program has a four-tiered structure that organizes the curriculum. First, one 
signature of Vanderbilt’s leadership preparation programs and a characteristic that separates us 
from other competitor institutions is the centrality of learning to our degree. It is the touchstone 
from which our programs begin. Second, from this understanding of learning, the curriculum 
extends to the social and policy contexts in which education occurs, the organizational dynamics 
at work in complex institutions, and the leadership theories one can use to examine institutional 
change. There is a focus on the way in which organizational leaders can create change and propel 
a school or a university toward a shared vision. The third tier of the curriculum is comprised of 
courses designed to equip students with the skill set necessary to design organizationally based 
research questions. Different from a traditional research methods sequence, the courses focus 
on how problems might be defined so that analytic tools may be brought to bear on identified 
organizational problems. The fourth and final tier contains courses focused on the specialized 
knowledge needed in a participant’s focus organization, either K-12 or postsecondary. (For a 
more detailed examination of the Vanderbilt curriculum, see Goldring & Schuermann, 2009, in 
this issue of Peabody Journal of Education.) 


Capstone Project 


The capstone project replaces the traditional dissertation as a culminating experience for the 
program as a method of certifying competency. We used a consultancy model as the guiding 
concept for creating the capstone experience because consulting requires many of the skill sets 
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expected from educational leaders and it addresses a substantial and authentic problem of practice 
as identified by an organization. 

The final goal of the project is that a team of two to four students will present meaningful 
recommendations to the client organization and the faculty. Student teams must negotiate entry 
into the organization, which requires an intense understanding of organizational theory and 
the literature, including how organizations work, politics, social context, institutional hierarchy, 
norms and power. Additionally, the project requires that students have an understanding of the 
relevant literature, which is demonstrated through a targeted literature review that is used to frame 
questions and evaluate results. The reviews are narrower than what one would find in a traditional 
dissertation. Also, the capstone employs both quantitative and qualitative methodologies to answer 
questions posed by the teams. Though there is not usually a contribution to the knowledge base 
from the capstone, because it is more focused on client needs, there are contributions to the 
broader community of practitioners. (For a closer look at the Vanderbilt Capstone Project, see 
Smrekar & MGraner, 2009, in this issue of Peabody Journal of Education). 


Student Performance Appraisal 


Student performance is evaluated through three processes: course grades, comprehensive exami- 
nations, and the capstone project. Conventional individual course performance measures (such as 
grades) are coupled with annual reviews by program faculty who provide performance feedback 
and a review of participants’ progress along with formative recommendations for improvement. 

In addition, each participant must successfully pass a written, closed-book qualifying exam- 
ination. The purpose of the written examination is to evaluate students’ levels of mastery and 
ability to synthesize subject matter presented in the initial 2 years (six semesters, 36 credit hours) 
of the program, including their capacity to apply this knowledge and these skills to practical 
settings and problems. It is a 6-hour, closed-book exam. 

A program participant is initially eligible to sit for the examination following completion of 
the first 12 scheduled classes. The examination is offered on a Friday in May of Year Two. The 
examination begins at 8:30 a.m., breaks for 60 minutes, and resumes from 12:30 until 3:30 p.m. 
Illustrative questions and problem sets are distributed during the preceding January. A second 
administration of the examination is offered, assuming faculty concurrence, 3 months later, in 
August, for any participant not gaining a passing grade upon initial administration. 

Program participants are granted doctoral candidacy upon successful completion of the quali- 
fying examination and successful completion of the fourth research methods course. Candidacy is 
granted for only one calendar year, during which individual students are expected to successfully 
complete the capstone experience. The capstone serves as an opportunity to integrate and use 
skills gained during the course of the program. 


DIFFERENCE BETWEEN THE RECONCEPTUALIZED Ed.D. 
AND THE Ph.D. PROGRAMS 


With the program revision complete, the final product is substantially different from the Ph.D. (and 
from most other Ed.D. programs). This differentiation serves as a signal to multiple marketplaces. 
Specifically, it serves as a model of what a practice-oriented terminal degree should look like 


68 T. C. CABONI AND E. PROPER 


for aspiring education leaders; it suggests that the most appropriate degree for those interested 
in practice is the Ed.D., not the Ph.D.; and it demonstrates to potential employers that for 
administrative positions a well-conceived Ed.D. may be more relevant than a Ph.D. 

At Vanderbilt, the Ed.D. aims to prepare individuals with an orientation to practice, whereas the 
Ph.D. primarily prepares individuals interested in becoming faculty members at research univer- 
sities. The Ed.D. is designed for mid-career professionals who seek a terminal degree and career 
advancement within their organizations, whereas the Ph.D. prepares junior faculty for their initial 
professional positions in the academy. The Ed.D. is oriented toward solving problems of practice, 
whereas Ph.D. students engage in an immersive research program following an apprenticeship 
model, focusing on advance research methods and knowledge creation. The Ed.D. takes as its 
goal placement of graduates into senior leadership positions within educational organizations; 
the Ph.D. program places its graduates into faculty lines at peer research universities. 
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The Ed.D. program at the University of Southern California focuses on leadership in urban education. 
The article describes the rationale and guiding principles for the program and its design in terms of 
core knowledge, inquiry, signature pedagogy, concentrations, and capstone experiences. The article 
concludes with an assessment of our contributions to date and a set of challenges that will direct our 
efforts in the near future. 


The purpose of this article is to describe a new Ed.D. program to educate educational leaders to 
solve problems they face in their work setting. We began our design of the new Ed.D. program 
with a concept that would lead us in a direction very unlike that of traditional Ph.D. programs. 
Instead of starting with the history, theory, and research in different foundational disciplines, we 
identified the tasks that leaders face in different educational contexts. Our goal was to focus on 
what our students should be able to do as professionals in the new education-reform contexts 
and on our mission in urban education: We wanted to connect a huge amount of conceptual and 
clinical knowledge with best use of research and best practices to produce effective leaders for 
urban education. For us, knowledge would integrate all courses and would center on what students 
need as professionals. We also wanted a common problem-solving model, one that could serve as 
a signature pedagogy for our program’s core, concentration, and culminating, or capstone, work. 


ENVISIONED PROGRAM CHARACTERISTICS 


For program governance, we envisioned a process that would link the authorities and responsibil- 
ities of the dean, associate dean, and faculty council. The new Ed.D. program would be led by a 
faculty member who worked full time as the program director. A program governance committee, 
composed of faculty members, would provide curriculum and program guidance. We also wanted 
much greater faculty collaboration in ways that fit with the overall program governance strategy, 
as well as a strong and formalized student advisory process that offered clear and collaboratively 
developed written program guidelines for both faculty and students. 
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Further, we wanted the new program to have effective, ongoing oversight and review: data- 
based decision-making and problem-solving, as well as active use of indicators, data, and mon- 
itoring of success and problems that would continue to improve the program. We planned to 
initiate periodic reviews of our work, which would include evaluations by external visitors and at 
the formal program level. For curriculum development, we sought collaborative faculty involve- 
ment. We wanted to shift from specific courses being “owned” by specific faculty members to 
all courses being collectively developed and thus owned by the program. And we wanted each 
developed course to have one syllabus that all faculty members would use. 

In our vision of the new Ed.D. program, each Rossier School of Education (RSOE) faculty 
member would see him- or herself as an Ed.D. program faculty member. We planned to build a 
new culture of respect and mutuality among faculty members that would lead to new levels of 
collaboration. We were aware of the advantages of such a culture, but we also knew it would 
foster dilemmas and problems as well. As a result, we wanted the idea of “faculty” to be expanded 
to create equal status for and engagement of all full-time faculty, whether tenured/tenure-track 
or clinical. Additionally, adjuncts and other practitioners currently in the program were to have 
an enhanced role in the expansion process. We looked at this idea as a way to help create our 
desired new connection with the field: helping faculty draw on their strengths and assets within 
the program. 

The envisioned program was also going to have a new organization, new leadership, and new 
support for students. Our view was that we needed a skilled administrator to lead the program 
and good support services to assist the faculty and staff in implementing the program. 

We wanted the new program to have much more consistency than the previous one had. New 
students were to start the program together as a cohort each fall, and there would be much stronger 
support for students. There would be an Ed.D. program office instead of a department structure, 
and it would have a strong student-support staff so that students would be able to solve a wide 
array of academic dilemmas at one location. We also wanted to dramatically improve student 
recruitment—to build on our field connections and our tradition of mentoring—by admitting 
leaders with the insights and strengths of the diverse urban setting we wanted to serve. This effort 
would also include recruiting students with high academic qualifications. Revitalizing our career 
placement center and career mentoring would be integral to this overall process. 

We envisioned the new program as offering a strong set of core courses that all students would 
take. This core would feature powerful levers for effective practice—leadership, accountability, 
diversity, and learning—all of which came out of our 2001 futures conference as academic 
themes. The set of concentrations would expand to include such job-related concentrations as 
K-12 and higher education/community college, offering students some choice and focus. These 
concentrations would provide a meaningful yet limited set of options that were administratively 
feasible for our faculty and management approach. We also wanted to identify specific criteria 
for determining these concentrations, a task we knew would be controversial for the faculty. 

For the culminating, or capstone, experience, we wanted the program to take an entirely 
different approach. Instead of the traditional experience of addressing gaps in the theoretical 
literature, our capstone experience would address relevant problems of practice. Through this 
experience, students would be helped to make a difference in their professional practice both 
during the program and, especially, afterward. They would also work collaboratively, as they 
would throughout their careers, and they would learn how to inquire and use best practices to 
address real-world problems. We saw the changeover to this new capstone experience being made 
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in stages rather than immediately, for two reasons: the university’s concerns about quality and 
the faculty’s diverse views and lack of experience with such an approach. 

In contrast to the prior Ed.D. program’s structure, in which students could (and did) take 11 
core courses in essentially random order, the new program was to be structured so that the first 
year consisted of the four core courses for all students. In addition, the second year would build 
on what students had learned in the first, and the core and concentration courses would be closely 
connected. In short, the program would have layers that built on each year’s work. The program 
would also have new approaches to assessment and student accountability. We were determined 
to have more student accountability than previously and a requirement for steady progress in the 
program—to avoid creating a legacy-student problem akin to the one inherited from the previous 
era. 

As another avenue for improving the Ed.D. program, we sought to collaborate with other 
professional programs and to draw on the best thinking about professional doctorates that was 
available. We sought to be part of the national Ed.D. leadership while also recognizing the views 
of our alumni and other local stakeholders. Additionally, we were aware that we needed external 
credibility and validation—and that we needed to deserve them. Given the range of views within 
the RSOE faculty and the demand for excellence across USC, we knew that our interest in 
enhancing professional practice and urban education for our students would require external 
support. 

Finally, we knew that in setting out to enact our vision, we were embarking on a bold and 
important adventure that was likely to have difficulties. But we also knew we could not turn back: 
It was time to move forward with the new Ed.D. program. 


Our View of Urban Education 


We view urban education as the process of teaching and learning that takes place in complex 
urban-metropolitan settings typically characterized by broad diversity in race, ethnicity, gender, 
class, culture, and language abilities. Urban settings have a mature service-delivery infrastructure 
characterized by sociopolitical stratification and unequal access by citizens to services. 

When we speak of the study of urban education, we mean the critical examination of the 
manner in which issues of broad diversity may intersect in complex ways to marginalize and 
adversely affect the learning and teaching process for some populations. A key element in our 
vision of urban education is a commitment to function not just as observers or commentators but 
as change agents, in our research, teaching, and service. This commitment presumes the creation 
of a collaborative learning community in our urban setting that advances knowledge about the 
skills and tools needed to enact social change in the pursuit of social justice—that is, to connect 
education practice to critical inquiry in urban settings. To this end, the urban paradigm played a 
key role in the content of all courses set forth as central to the new Ed.D. program. 


The Four Academic Themes 


Part of what we did at the conference was to identify the intellectual framework of academic 
themes that would support our urban mission. The themes had to integrate the insights of 
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academic disciplines and clinical knowledge in ways that made professional knowledge useful 
to practitioners; they also had to frame our research efforts in ways that supported professional 
improvement. We identified four such academic themes: leadership, accountability, diversity, and 
learning. 


Distinction Between Ed.D. and Ph.D. Programs 


Another outcome was our achievement of a clear distinction between the Ed.D. and Ph.D. 
programs offered by the RSOE. Extensive changes had already been made to the Ph.D. program’s 
design prior to 2000, primarily in response to pressure from USC’s central administration to 
offer only high-quality Ph.D. programs. We thus were at a point where we could design a 
new program that would be distinct from and of equal quality to the Ph.D. program. We saw 
both programs as emphasizing the acquisition of appropriate research and inquiry skills, but 
these skills would be applied differently. The Ed.D. student would be trained to use education 
inquiry skills to solve contemporary education problems; the Ph.D. student would be trained 
to contribute to the general and theoretical knowledge about education issues. Thus, the Ed.D. 
student would engage in research and development efforts aimed at directly improving education 
practice while the Ph.D. student would aim to contribute to theory and general knowledge in a way 
expected to be relevant to solution of a problem but not necessarily applied to current education 
practice. 

In a recent issue of the Educational Researcher, Shulman, Golde, Bueschel, and Garabedian 
(2006) point to the RSOE as a rare “good example” of a school of education that has made a strong 
and appropriate distinction between the Ph.D. and Ed.D. degrees. Shulman makes an important 
point when he states that it is difficult to move forward with a new Ed.D. program without also 
addressing the need to refine the Ph.D. program. 

Our new Ed.D. program—the result of which is a premier degree in the RSOE—is specifically 
designed to produce practitioner leaders who are expert in promoting the enhancement of learning 
within an organization, an expertise that is relevant both in schools and in all other institutions 
where learning occurs. 


DESIGN APPROACH AND PRINCIPLES 


The Ed.D. planning committee began discussing the design of a new Ed.D. program at the 
start of the fall 2001 semester. This committee consisted of five tenured professors and four 
clinical professors. Myron H. Dembo, a tenured professor of educational psychology, and Stuart 
Gothold, a clinical professor of education and a past superintendent of the Los Angeles County 
Office of Education, were chosen to serve as cochairs. They and the seven other faculty members 
represented different academic divisions of the RSOE. 

The planning committee approached its task by instituting an open process. It invited all faculty 
members to present their concerns and recommendations throughout its deliberations. Faculty 
members simply had to contact one of the cochairs to be placed on the agenda for any of the 
committee’s monthly meetings. This approach greatly reduced the possibilities for isolation and 
conflict, because when the time came for the whole faculty to vote on the new program, almost 
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every faculty member had taken advantage of the opportunity to voice his or her concerns. The 
result was overwhelming approval of the new program at the end of the spring 2002 semester. 

Faculty members praised the approach used for planning and curriculum development for being 
a professional-development opportunity the majority of them had never before experienced. They 
reported that they had never spent so much time discussing, debating, and evaluating their views 
about course content within a committee environment. This is not to say that the process was 
always harmonious; difficult times did indeed occur. However, in the end, an innovative Ed.D. 
program was developed. 

The planning committee adopted the following guiding principles for the design of the new 
Ed.D. program: 


e To differ from the Ph.D. program, the new Ed.D. program will involve students in activities 
that strengthen their leadership skills throughout the program. That is to say, the program 
has to be based on the question: What cognitive and interpersonal knowledge and skills 
are necessary to function successfully in leadership positions in education? Our former 
program focused on the question: What knowledge do graduate students need to acquire in 
educational psychology, sociology, policy and administration, and other broadly defined 
academic disciplines? The new program will not stop emphasizing academic content, but 
its starting point will be educational settings rather than academic content. Our belief is 
that students’ weekly academic experiences should be related to competencies and skills 
used in their chosen professions. 

© Clinical faculty members will have equal status with tenured/tenure-track faculty members 
in developing the program. This principle ensures that we will benefit from the opinions 
of some of the best practitioners and clinical scholars on the question of what skills and 
competencies education leaders have to develop. Our guiding belief is that the change 
we want to implement cannot be accomplished without the input and involvement of the 
clinical faculty. 

e Instructional processes will focus on problem solving, discussions, and small-group work 
rather than the traditional lecture format. This is the principle that we have adapted for 
our signature pedagogy, which is a problem-solving model based on a gap analysis of 
practitioner challenges and goals, current situations, and strategies (see Clark & Estes, 
2002). 

e A core curriculum will be established for the first four courses. Both these core classes 
and the ensuing concentration courses will have common syllabi, and all students will take 
courses in a specified sequence. Concentration courses will build on the interdisciplinary 
knowledge gained from core courses. Our goal here is to provide students with a common 
course experience regardless of who teaches the course. In addition, this will ensure that 
instructors know what knowledge students have acquired in previous classes. 

e Students will complete the program in 3 years. This principle led to an administrative 
structure that included an Ed.D. executive director with dedicated advisors for each new 
doctoral class. In the past, students were in the program anywhere from 3 to 10 years, 
taking courses in random sequence with different instructors who taught the same course 
but chose to cover different material. This structure made it nearly impossible to determine 
students’ standing in the program, their level of expertise, and whether their status was 
active or inactive. And, the coupling of this structure with the cumbersome and inadequate 
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record-keeping systems that existed made it extremely difficult to identify, track, and 
mentor students in the program and to develop accurate yearly budgets to support the 
program. 

Students will stay with their entering cohort as they move through the program. Because 
collaboration is such an important skill in most education positions in the real world, we 
wanted to model this element during the Ed.D. program. 

Over 80% of students in the program will participate in thematic, rather than traditional, 
dissertations. Each student will research and write his or her own dissertation, but all 
student work within a cohort will relate to common themes or education issues and 
challenges and will be done in small groups of students directed by one or two faculty 
advisers. We chose this form of capstone experience because it fosters and is based on the 
skills of inquiry and collaboration that we want graduates to carry with them as education 
leaders. 


THE NEW PROGRAM 


This section provides the details of our new Ed.D. program’s design and implementation, our 
experiences along the way, and related issues and challenges. We start with an overview of the 
3-year program as context; from there on, the discussion is organized by the major elements of 
the program: the core courses, concentration courses, inquiry methods courses, and culminating, 
or capstone, experience. 


Overview 


All students admitted to the Ed.D. program must have a master’s degree and at least 3 years of 
relevant work experience. Table 1 shows the 3-year course sequence established for earning an 
Ed.D. 

Each student begins the program in the fall semester, entering as one member of a cohort. 
All students in a cohort take the same four core courses in the first academic year. The follow- 
ing summer, they take the first of two inquiry methods courses and begin work in their choice 
of one of four concentrations: (a) Educational Psychology (applying teaching and learning in 
schooling/business and nontraditional educational settings), (b) Higher Education Administra- 
tion/Community College Leadership, (c) K-12 Leadership in Urban School Settings, and (d) 
Teacher Education in Multicultural Societies (TEMS). Work connected with the thematic disser- 
tations begins in the students’ first summer and continues to the end of the program. 


Core Courses 


The conceptual framework from the 2001 futures conference became the conceptual base for 
our Ed.D. core program. That is, the four academic themes underpinning our urban education 
mission—leadership, accountability, diversity, and learning—became the basis for our four core 
courses. Along with acknowledging the benefits of having all Ed.D. students take a core set 
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TABLE 1 
Three-Year Sequence for Ed.D., by Year and Semester 
a 
Year 1 
Fall: 
Challenges in urban education: Leadership (Educ 524) 
Challenges in urban education: Accountability (Educ 522) 
Spring: 
Challenges in urban education: Diversity (Educ 523) 
Challenges in urban education: Learning (Educ 525) 
Summer: 
Summer conference 
Inquiry methods I (Educ 532) 
Concentration course 1 
Year 2 
Fall: 
Concentration course 2 
Concentration course 3 
Spring: 
Inquiry methods II (Educ 536) 
Critique of research (Educ 792) 
Summer: 
Concentration course 4 
Concentration course 5 
Dissertation research seminar (Educ 790) 
Qualifying exams 
Year 3 
Fall: 
Dissertation 
Spring: 
Dissertation 
Graduation 





of courses, we thought it imperative that core content be generic so as to apply equally to all 
students regardless of concentration. There were several good reasons for having the core courses 
mirror the conceptual framework: the four elements provide powerful guidelines for practitioner 
leadership, they help integrate key theory relevant to practice, and they reflect important work 
strategies and dimensions of the actual work that leaders need to implement. 


Rationale for Four Core Courses 


The four core courses proposed by the planning committee for the Ed.D. program and their 
rationales are described in the following paragraphs. 


Leadership. This course covers “how” to focus education systems on learning. The goal 
of the course is to produce leaders who are committed to and capable of transforming ur- 
ban schools into high-performance learning organizations. To this end, the course focuses on 
enhancing the skills and knowledge of people in the organization, creating a common culture of 
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expectations, fostering productive relationships within the organization, and holding individuals 
accountable. Students also learn how to examine the organization in which they work and how to 
create improvement strategies that pay attention to the structural, human relations, political and 
symbolic/culture dimensions of their organizations. 


Accountability. The accountability course helps students develop indicators of success in 
educational organizations, especially for determining what should be learned and how well it has 
been learned. Students then learn how to use a problem-solving model to analyze performance 
“gaps,” how to search for evidence-based best practices, and how to use data to drive organizational 
improvement. The course also helps students address accountability issues related to leadership, 
diversity, and ethics in organizational change and how to establish a culture of continuous 
improvement and organizational learning in their settings. 


Diversity. This course deals with the context within which educators operate, particularly 
in urban areas. It promotes an understanding of the specific strengths and needs of learners who 
differ in income, ethnicity, gender, language proficiency, or disability and ensures that graduates 
incorporate what they learn here into their practice. 


Learning. This course presents the basic principles of how individuals and groups learn and 
teaches individuals how to apply a problem-solving model to solve educational problems. The 
course focuses on the application of learning principles in different social contexts. 

An important goal in planning the core courses was to integrate all four themes whenever 
possible. For example, we wanted the diversity course to cover relevant topics on leadership, 
accountability, and learning. In addition, it was our intention to have the concentration courses 
build on what students had learned in the core classes. This strategy provided students with 
important foundational skills for later, more advanced work and allowed teachers of later courses 
to know what knowledge their students had already acquired. 

Here is what a faculty member said about how the sequence of core courses helped her teach 
a concentration course in instructional leadership and school improvement: 


As I began teaching student engagement, I knew what the students covered in motivation in their 
learning course, so I was able to introduce topics knowing that they had the background. The same 
thing can be said about accountability. I am able to talk about different accountability measures and 
how data drives decision-making. I don’t have to deal with important issues like criterion and norm 
reference measures because I know they were covered. As I talk about student engagement and the 
leadership of principals, I don’t have to go back and survey what leadership is all about. I know what 
they covered in their leadership class. 


Core Course Foci and Development Issues 


Core Leadership Course. This course focuses a great deal on self-assessment and reflec- 
tion. Students use a number of self-assessment instruments and explore theories of leadership; at 
the end of the course, they summarize in a paper what they have learned about themselves. 
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Core Accountability Course. The members of the subcommittee developing the account- 
ability course had to deal with their different conceptualizations of accountability, especially the 
psychometric approach versus the policy and systems view. They then had to determine how 
to deal with accountability issues in different educational contexts and relate them to leader- 
ship, diversity, and learning. The course traces the evolution of the accountability concept, with 
an emphasis on emerging accountability issues in urban schools and colleges that serve large 
numbers of low-income and racial/ethnic students. Special attention is given to helping leaders 
understand, analyze, and cope within the context of increased demands for accountability. This 
course adopted the same problem-solving approach as the learning course did: gap analysis. 


Core Diversity Course. This course focuses on helping students clarify their own thinking 
about diversity issues, identify problems, and consider solutions. The focus of this course took 
some time to arrive at, however. It initially ran into difficulty because a few faculty members who 
were developing the course believed its content focused more on problems than on solutions. Over 
time, the subcommittee resolved this issue by identifying multiple frameworks for addressing 
the achievement gap and recommended suggestions for implementing change. For example, one 
suggestion was to have each student use an Internet discussion board to make an original entry 
each week that addresses one or more of the readings and analyzes that he or she has read. Each 
student then responds to two other students’ entries on the discussion board. Faculty members 
would not respond to the student discussion but would frame questions to clarify issues in the 
next class meeting. 


Core Learning Course. The members of the subcommittee developing the learning course 
agreed on a gap analysis problem-solving approach for the course early on (see Clark & Estes, 
2002). In this approach, students define goals, determine the gaps by comparing the goals to 
the standard used, determine the causes of the gaps (i.e., knowledge, motivation, and/or cul- 
ture/context), determine solutions for closing the gaps, and develop an evaluation plan for the 
recommended solutions. The subcommittee members debated the significance of different read- 
ings when they initially attempted to make the course relevant to students in K-12, higher 
education, business, and human performance. They started with a textbook in the first year, 
moved to a book of readings in the second year, and then returned to a textbook for limited 
readings plus two books, one on gap analysis and one on goal setting. 


A Signature Pedagogy 


Since our program focuses on helping educational leaders to develop problem-solving skills they 
can use in their jobs, we introduced a problem-solving model call gap analysis (see Clark & Estes, 
2002). The model was adopted in two of the four core courses (i.e., accountability and learning) 
and is the focus of the education psychology concentration. The faculty is currently debating its 
use in the overall Ed.D. program. 

The key approach to gap analysis is that it asks leaders to answer the following questions as 
they attempt to solve performance problems: 


1. What is our performance goal? 


78 D. D. MARSH AND M. H. DEMBO 


Where are we now (related to goal)? 
What is the size of the gap? 

What is causing the gap? 

What solutions will close the gap? 
How do we implement the solutions? 
How do we measure our progress? 


aT ON, Se AD 


In the learning course, for example, the students acquire knowledge regarding each of the seven 
steps and conduct a case study based on a problem that they face in their work environment. They 
do not collect actual data but explain how they would use the steps to solve their problem. Thus, 
students learn how to write goals and assess achievement gaps, diagnose causes of performance 
gaps, design and test gap solutions, and evaluate outcomes. They also learn that there are three 
major causes of performance problems: knowledge (the why, where, and when), motivation (when 
we don’t want to do it, think we can’t, or don’t value it), and organizational policy (when we are 
prevented by policy, procedures, culture and/or lack facilities and equipment). Students use the 
literature to inform their analyses of the causes and the solutions they plan to implement. 

The rationale for gap analysis stems from the evidence that many leaders fail to analyze the 
causes of performance gaps in that they (a) often fail to have clear goals before they embark 
on finding solutions to problems, (b) often select and implement the wrong solutions, and/or (c) 
when solutions do not work, they often blame the people who have the problem. 


Inquiry Methods Courses 


The inquiry methods courses have been the most difficult part of Ed.D. program development. 
There are three possible purposes for inquiry methods courses in a doctoral program: to (a) 
enhance the professional competence of students by promoting an identified set of important in- 
tellectual skills necessary for effective education leadership, (b) prepare students for the academic 
requirements in their course work, and (c) prepare students to conduct research investigations 
required by their thematic or individual dissertations. 

We decided to use the two inquiry methods courses to enhance students’ professional com- 
petences by promoting an identified set of important intellectual skills necessary for effective 
education leadership. Students with these skills will be able to: 


Make valid inferences from qualitative and quantitative evidence 

Use and do scientific research in everyday educational practice 

Analyze education programs, policies, institutions, and processes 

Evaluate education programs, policies, institutions, and processes 

Create solutions to education problems using scientific evidence to drive decision-making. 


What we want from our two inquiry methods courses is for students to approach their challeng- 
ing workplace problems using rigorous inferential thinking, in much the same way that scientists 
use inferential thinking to solve theoretical problems. 

Currently, the inquiry methods courses are not fully developed. Once they are, we plan to 
have all new students begin the program with Inquiry Methods I. Students will then be better 
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prepared for the academic demands of the four core courses. Both inquiry methods courses are 
meant to assist students in their dissertation research, but we do not believe they provide sufficient 
background on using research methods for collecting and analyzing data. For this reason, we are 
developing separate online modules on collecting, analyzing, and using test data and collecting, 
analyzing, and using interview and observation data. 

This year we formed a task force to modify the recently developed course syllabi. Next 
month we are conducting two focus groups of leaders in both the K-12 and higher education 
environments. Our goal is to learn more about the specific inquiry tasks/functions that leaders deal 
with in their work settings. We will use this knowledge to guide us in our curriculum changes. 

Our current thinking regarding the course revisions involves beginning the first inquiry course 
with two case studies that identify issues in learning, diversity, accountability, and leadership. 
In the appendices of the case studies would be documents that include data from school dis- 
tricts/community colleges that would need to be analyzed as the students work on the case 
studies. The course would provide students with inquiry tools to collect and analyze data to solve 
problems. Statistical information would be presented to help students use the inquiry tools. We 
would determine how each of the core courses could use the case studies as an advance organizer 
for presentation in the first-year courses. 


CAPSTONE EXPERIENCE 


As the capstone experience of the Ed.D. program, the student is required to verify his or her 
ability to inquire at the doctoral level. Historically, students have had to fulfill this requirement by 
conducting original research in the form of a traditional dissertation. Because we envisioned our 
new Ed.D. degree as being more related to the students’ specific professional position or future 
goals than it had been in the past, we wanted the Ed.D. dissertation to result in improved education 
practice. Our preferred path was thus the thematic, rather than the traditional, dissertation. 

The ideal way to proceed with the change to thematic dissertations would have been to simply 
require all dissertations in this form. However, issues of quality at the university level and the 
unfamiliarity of many faculty members with this new form of dissertation cautioned us about 
making such a radical change in one step. We decided to approach this issue in stages, the first one 
being approval of two Ed.D. dissertation paths, the traditional and the thematic. Our established 
goal was that 80% of student dissertations would be thematic. 


Assets of the Thematic Dissertation 


There are distinct differences between these two forms of dissertations. The traditional dissertation 
requires the student to work closely with a faculty member to conduct a research investigation 
and produce an original and unique dissertation. This path is virtually identical to that pursued 
by Ph.D. students. This form of dissertation requires faculty members to devote large amounts 
of mentoring time to students on an individual basis. The bulk of the dissertation is written alone 
and without benefit of peer support. 

Like the traditional dissertation, the thematic dissertation results in an original and unique 
dissertation from each student. It differs, however, in that several students work on related topics 
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or with the same database. In addition, the themes for the dissertations are generally organized 
around field-based issues or problems and require students to collaborate with each other as they 
develop their proposals and to critique each other’s work. In short, there is a unifying feature that 
ties some of the Ed.D. students together such that they can be mentored as a group. 

Another key difference with thematic dissertations is that students begin with a problem and 
then analyze the literature for guidance on how to research the problem, rather than engaging 
in the typical traditional dissertation process of reviewing the literature to identify gaps and 
constructs and then deciding on a setting for conducting their research. Although the thematic 
dissertation mode requires a great deal of faculty time, the total time devoted to the group is 
less than what would have been accorded had each student in the group chosen the traditional 
dissertation mode. Another feature of the thematic dissertation mode is that it allows a faculty 
member to design the group in accordance with her or his area of expertise and current research 
agenda. 

Students participating in thematic dissertations meet both formally and informally in all stages 
of their projects. This model of collaboration is apparent throughout the dissertation process. 
Students can assist each other in every aspect of the task and can critique and learn from each 
other’s efforts. As a result, the thematic dissertation group increases individual productivity and 
accountability and can produce robust studies that may make a significant contribution. 

For the most part, the graduate students in our program are full-time professionals who com- 
plete their program of study in the afternoon and evening. The thematic approach to dissertations 
serves many purposes for these students. First, it encourages collaboration, which is related to the 
tasks that students experience on the job. Second, it emphasizes inquiry training around practice. 
Third, it is more concerned with helping these practitioners deal with field-based problems and 
issues than with initiating them in the work of academic scholars, which is not the primary focus 
of the majority of students. Fourth, thematic dissertations have the potential to provide important 
research data for school districts and higher education institutions. The traditional, individual 
dissertation is offered as a viable option for those students who plan to teach in an institution of 
higher learning. 


Thematic Dissertation Groups 


The following paragraphs describe some of the thematic dissertation groups that have been formed 
in the program. 


Large-Scale Education Reform: Assessing the Evidence. This group, which was di- 
rected by Professor Amanda Datnow, focused on “large-scale educational reform in the era of 
accountability.” The movement toward large-scale reform marks a shift away from the notion 
that school improvement should proceed organically, one school at a time. Rather, the idea is 
to achieve change across a large number of schools at the same time. Examples of large-scale 
reform include the federal No Child Left Behind Act and the nationwide scale-up of the small 
schools model. 

The students in this dissertation group conducted qualitative studies on various aspects of 
large-scale education reform in K-12 urban settings. The methods were “held constant” while the 
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students chose their own topics and conducted completely different studies under this broader- 
topic umbrella. The dissertations clustered into various subgroups within large-scale reform, 
including leadership, data-driven decision making, and curriculum implementation. 

For example, one student’s dissertation explored how Reading First programs are implemented 
differently across schools and why. Even though the two schools in this study had differences 
in achievement levels, there were no marked differences in implementation and level of district 
support. However, there were differences in leadership stability that may have contributed to 
differences in achievement across the two sites. 

A second student explored how standards-based instruction (SBI) is implemented in 
mild/moderate special education classrooms. He found that though teachers were eager to em- 
brace the standards, they lacked the materials and resources to do so. Therefore, even though they 
used standards to guide their instruction, those standards tended to be several years below grade 
level. 

A third student compared the philosophies and ideas of traditional and nontraditional super- 
intendents in urban school districts. Superintendents were interviewed about their perspectives 
on accountability measures pertaining to federal mandates, community involvement, and school 
board relations. 


Effective Practices in High-Performing Charter Schools. In another thematic group, 
this one directed by Professor Priscilla Wohlstetter, students conducted case studies of promising 
practices in California charter schools to document strategies that effectively support student 
achievement. Each student in the group chose one of the following main content areas as her or 
his focus: 


Improving administrative and teacher leadership 

Creating school—university partnerships 

Improving high school reform efforts 

Increasing parent involvement 

Improving student behavior 

Teaching through project-based learning 

Improving literacy for English-language learners 

Teaching through the arts 

Improving special education 

Improving math and science education through the use of technology 


Promising practices were identified through a statewide nomination process, and 20 of the 
practices were ultimately selected for the compendium. Each student was responsible for inves- 
tigating two promising practices (at two different schools) in his or her content area. 

Students collected information during site visits to the charter schools, primarily through 
interviews and focus groups with principals, teachers, and parents. Findings from the students’ 
dissertations were subsequently incorporated into USC’s Compendium of Promising Practices, a 
searchable online database developed by the Center on Educational Governance. The compendium 
is an ongoing project; every other year, a new cohort of students will conduct field research 
on promising practices in California charter schools, the results of which will be added to the 
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compendium. (For details on the compendium, please visit the Center on Educational Governance 
web site at http://www.usc.edu/dept/education/cegov/.) 


PROGRAM EVALUATION AND REPUTATION 


We have collected and analyzed extensive data about our new Ed.D. program and have used 
those data to redirect and refine almost every aspect of the program. The data were gathered both 
internally by the RSOE and externally by a research group, Vital Research, LLC. We have also 
paid attention to some external perspectives that have provided us with useful feedback about our 
work. 


EVALUATION OF FIRST CYCLE OF PROGRAM 


In 2009, we will complete an evaluation of the first cycle of the new program. This evaluation will 
focus on the adequacy of the original program design, the extent to which the original program 
was effectively implemented, the views of graduates and other stakeholders about the program 
and its effect on graduates, the quality of the program from both a national and an internal 
perspective, and the challenges we must address in moving forward. 


NATIONAL ATTENTION 


The new Ed.D. program has received attention on the national level from both scholars and 
practitioners. The RSOE is currently regarded as offering one of the top Ed.D. programs for 
various concentrations. Scholarly articles, external market surveys, and national rankings serve 
as evidence of the external perceptions of the RSOE and its Ed.D. program. In 2006, U.S. News 
& World Report ranked three RSOE programs in the top 25 nationally: Higher Education ranks 
Sth in the nation, K-12 Leadership is 13th, and Education Policy is 21st. Our student body also 
has strong ethnic diversity as described in Borden & Brown (2006). 


Scholarly Articles 


Shulman et al. (2006) highlighted our Ed.D. program in an article focusing on the theoretical 
distinctions between the Ed.D. and Ph.D. programs. The authors proposed that if a school wanted 
to differentiate between its doctoral programs, the first step should be to clearly define the Ed.D. 
capstone experience. Starting from the final vision of what knowledge and skills a graduate of a 
professional doctorate program should possess, a university can begin effectively developing a 
specific pedagogy for imparting that knowledge and those skills. 

Shulman et al. (2006) also named the RSOE as one of the schools that has successfully 
transformed their programs. Their article describes the 3-day strategic planning meeting (our 
futures conference) in which the RSOE faculty convened to discuss and ultimately define its 
vision of the Ed.D. and Ph.D. programs. The Ed.D. program that emerged from this meeting 
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and subsequent work was a 3-year program clearly distinct from the Ph.D. program in purpose, 
student preparation, and time to degree. Development of the new Ed.D. program engaged the 
RSOE faculty, drew on expertise in the field, and continues to be a work in progress as the school 
strives to offer top-level preparation for education leaders. 

Murphy and Vriesenga (2005) called the RSOE Ed.D. program one of the doctoral programs 
that have clearly distinguished themselves from Ph.D. programs. Out of 161 Ed.D. programs 
offering a degree in Educational Leadership, 4 were cited for having created alternative disser- 
tations and “‘professionally anchored capstone” experiences: St. Louis University, USC, UCLA, 
and University of Pennsylvania. Each of these programs has addressed issues of program design, 
institutional change, creation of signature pedagogy and capstone assessments, and related efforts. 
The authors describe how the RSOE Ed.D. program has students producing dissertations focused 
on “meaningful topics” aimed at helping students gain “real world experience.” As a result, the 
authors state, the dissertations present “implications for practice.” According to the authors, the 
four schools they cite are models for other schools wanting to embark on the same path. 


Market Analysis 


In 2004, the RSOE commissioned Vital Research, LLC, an external consulting company, to 
perform a market analysis to examine the school’s reputation and the quality of its academic 
programs compared with those of other schools of education. Vital Research conducted telephone 
interviews with deans of nationally ranked schools of education and with superintendents of urban 
school districts; it also took direct reports from staff members in other USC departments that 
interact with RSOE staff, faculty, and students. 

The overall impressions of the deans were positive about the Ed.D. program; many of them 
were impressed with the clear distinction between the two doctoral degrees offered by the RSOE. 
Specifically, many deans were interested in the innovative idea of the Ed.D. program’s thematic 
dissertations and were curious to see what the results would be. Superintendents who were 
interviewed cited reputation and the “RSOE network” as key strengths of the RSOE programs. 
Superintendents also commented that, as employers, they appreciated the mix of theory and 
practice provided in the Ed.D. program. Other USC staff members noted the school’s current 
leadership and its efforts to continually improve program quality as strengths of the RSOE. 


THE CHALLENGES AHEAD 


We have identified a set of challenges associated with improving and maintaining our new Ed.D. 
program that warrant attention. This section discusses these challenges. 


Inquiry Courses 


Although we have an applied focus for the inquiry methods courses in our Ed.D. program, issues 
remain about the nature and depth of knowledge students must acquire to use inquiry skills 
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effectively in their work environments and to complete their dissertations. As a result, we need 
to devote more attention to inquiry course development. 


Capstone Experience 


We are proud of our thematic dissertation approach but know that it may not be the best vehicle 
for our capstone experience in the future. We formed a subcommittee of our Ed.D. governance 
committee that is to develop options for what the capstone could look like, and we have obtained 
information from seven other universities about how their dissertations and/or capstones are 
accomplished. We need to consider several issues in designing the capstone experience: What 
makes more sense from the perspective of professional leadership enhancement? What is feasible 
and desirable from the perspective of program size? What is best from the perspective of program 
quality? 


Core Curriculum 


We are pleased with the extensive and highly collaborative process that led to the initial design of 
the four core courses, which have all now been implemented and revised over 3 years. We believe 
it is time to review the core courses with a view to determining how they are contributing to our 
overall goal of creating effective leaders in urban education. 


Signature Pedagogy 


We want to further evaluate our use of the gap analysis model, looking at it in terms of faculty- 
wide agreement about its role, ways to expand its use in the inquiry and concentration courses, 
and its proposed role as a required central feature of the new capstone strategy. 


Strategies for Enhancing the Ed.D. 


Considering the challenges of institutional change, we are pleased with the distinction in goals 
and intellectual activities that we have so far achieved in our Ed.D. and Ph.D. programs. Our 
goal is to direct our continuing development and implementation efforts toward establishing an 
Ed.D. program that will be nationally recognized as one of the premier doctoral programs in the 
country. To move forward with this goal, we joined the network of Ed.D. programs created by the 
Council of Academic Deans from Research Education Institutions (CADREI) and the Carnegie 
Foundation. With the help of this network, we hope to implement four related strategies for 
enhancing our Ed.D. program and meeting our goal of developing an outstanding professional 
doctorate for leaders in urban education: Coordinate through the Ed.D. governance committee. 
The governance committee meets monthly and deploys working subcommittees on important 
topics. For example, we are quite sure that the issue of the next version of the Ed.D. capstone 
will need to be fully discussed, leading to a formal proposal that the Ed.D. governance committee 
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will then take to the faculty council, the faculty, the dean, and ultimately the university-wide 
committee for approval. 


1. Engage in sustained program development once new program designs have been ap- 
proved. This development will be coordinated by the Ed.D. governance committee and 
the RSOE dean. The best historical example of this type of effort is the core curriculum 
committee, which was chaired by Professor Richard Clark. Over 18 months, teams of the 
faculty members comprising this committee worked extensively to develop the syllabi 
and instructional materials for each of the core courses from a general program design. 
For any new design—a new capstone design, for example—we will need this type of 
program development work. We also need a new round of improvement with our core 
courses. 

2. Take advantage of regional collaboration and impact. We envision a statewide network 
of Ed.D. programs and a conference at USC that would involve either CADREI/Carnegie 
and/or the regional institutions. We have already established a network with two private 
universities in Southern California that offer cutting-edge Ed.D. programs. This network, 
the Southern California Leadership Association, has a mission statement and 3-year goals 
that focus on two areas: networking with the best leadership programs internationally and 
supporting the enhancement of our Ed.D. program through critical friends, meetings, and 
evaluation. 

3. Seek out national collaboration and participate in the CADREI/Carnegie network. Na- 
tional collaboration and participation in the CADREI/Carnegie network will be vital 
to our efforts to improve our Ed.D. program and contribute to the national scholarship 
on professional doctorates. We intend to send a team to each of the CADREI/Carnegie 
Foundation national meetings and to participate seriously in the work of the network. 


This curriculum endeavor was not easy; even more importantly, it will never be completed. 
The faculty realizes that constant evaluations and adaptations are essential to maintaining the 
program. Nevertheless, we have clearly made extensive progress in 6 years and will continue 
to maintain a constant dialogue with members of the faculty and to use the Ed.D. governance 
committee to review, evaluate, and develop new policy for the program. 


REFERENCES 


Borden, V., & Brown, P. (2006). The top 100: Interpreting the data. Diverse Issues in Higher Education, 23(11), 34-103. 

Clark, R. E., & Estes, F. (2002). Turning research into results: A guide to selecting the right performance solutions. 
Atlanta, GA: CEP Press. 

Murphy, J., & Vriesenga, M. (2005). Developing professionally anchored dissertations: Lessons from innovative programs. 
School Leadership Review, 1(1), 33-57. 

Shulman, L. S., Golde, C. M., Bueschel, A. C., & Garabedian, K. J. (2006). Reclaiming education’s doctorates: A critique 
and a proposal. Educational Researcher, 35(3), 25-32. 


Copyright © Taylor & Francis Group, LLC 
ISSN: 0161-956X print / 1532-7930 online 
DOI: 10.1080/01619560802679740 


IN, 84: 86-99, 2009 8 
PEABODY JOURNAL OF EDUCATION, 84: 86-9 g Routled e 
8 Taylor & Francis Group 


A Professional Doctorate in Educational Leadership: 
Saint Louis University’s Ed.D. Program 


Susan Toft Everson 


Saint Louis University 


This article presents the case of a professional doctorate program in educational leadership at Saint 
Louis University (SLU). This Ed.D. program was designed to prepare students for executive leadership 
positions in school districts, distinguishing it from SLU’s Ph.D. program that prepares students for 
research or academic positions. This article describes the program’s background and theoretical 
foundation as well as its design and components. Concluding remarks focus on lessons learned 
during the program’s development and ideas regarding future work. 


This article describes the development and implementation of a professional doctorate program 
in educational leadership at Saint Louis University (SLU). It presents a description of the context 
in which the program exists; the background of the program’s development, including a brief 
overview of the theoretic foundation that grounds the work; the program design and components; 
and a conclusion that includes comments regarding lessons learned and future development. 


THE CONTEXT 
Saint Louis University 


Saint Louis University is a Jesuit, Catholic university ranked as a Carnegie research institution. 
Currently, the university has 12,733 students: 7,814 undergraduates and 4,919 graduate and 
professional students, including medical and law students. Students come from all 50 states 
and more than 80 foreign countries. Founded in 1818, SLU is the oldest university west of the 
Mississippi and the second oldest Jesuit university in the United States. A second campus is 
located in Madrid, Spain, where 637 of the students attend. 
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Educational Leadership Program 


One of SLU’s graduate departments is the Department of Educational Leadership and Higher 
Education, located in the College of Education and Public Service. The department offers a 
master’s degree and two doctoral degrees: a doctor of education (Ed.D.), the subject of this 
chapter, and a doctor of philosophy (Ph.D.), a traditional program that is designed to prepare 
students for research and academic careers. 


Student Population 


Currently in the Educational Leadership Program, 28 students are working on their Ph.D. degrees 
and 242 are working on their Ed.D. degrees. 

The doctor of education program focuses on preparing students for professional leadership 
positions. This program has grown significantly over the past 10 years. Unlike many other master’s 
and doctoral programs that prepare professionals at the preservice level, SLU’s doctor of education 
program primarily serves mid-career professional educators who are seeking superintendent 
certification. Over the past 5 years, the average ages of students enrolled in this program vary 
between 38 and 43 years of age, with an average experience level of 15 years working in education. 


BACKGROUND 
History and Rationale 


In the 1970s, the development of SLU’s doctor of education program was designed to match the 
students’ age and career status characteristics and to offer a leadership program that focused on 
the practical aspect of educational leadership. Initially, the Ed.D. doctoral project report was envi- 
sioned as substantially different from the research-based, five-chapter dissertation required of the 
department’s Ph.D. students. Over time, however, the Ed.D. doctoral project report evolved into 
a paper with all the characteristics of a Ph.D. dissertation. In early 2001, it became clear to SLU’s 
educational leadership faculty that this departure from the original intent of the Ed.D. project 
detracted from the problem-based orientation that they believed was essential to the leadership 
preparation. A decision to redesign the Ed.D. program was made to strengthen the problem- 
based learning format and to establish a professional doctorate approach to the preparation of 
educational leaders. 

Interestingly, the faculty’s work to rethink the design of SLU’s Ed.D. program coincided 
with a national conversation about the quality of educational leadership preparation programs 
(for example, see A. Levine, 2005; Murphy, 2002), the development of doctorates focusing on 
practice at other universities (for example, the University of Southern California and Vanderbilt 
University), and increasing support from scholars suggesting alternative approaches to doctoral 
education (for example, Golde & Walker, 2006; Maki & Borkowski, 2006; Scott, Brown, Lunt, 
& Thorne, 2004; Shulman, 2004; Walker, Golde, Jones, Bueschel, & Hutchings, 2008). 
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Program Approach and Key Decisions 


Early in 2002, the faculty members began a process to redesign their Ed.D. program to strengthen 
the problem-based learning format and to establish a professional doctorate approach to the 
preparation of educational leaders. From the beginning, the faculty articulated its intention to 
separate SLU’s Ph.D. program (as preparation for scholarship) from its Ed.D. program (as 
preparation for practice). This approach has been supported by Shulman, Golde, Bueschel, and 
Garabedian (2006), who described the Ed.D. as a “professional practice doctorate,” distinct 
from the Ph.D. which emphasizes instead “the techniques, habits of mind, and abilities of good 
researchers” (p. 28). 

SLU’s redesigned Ed.D. program includes components that replicate and model experiences 
that students will face in educational leadership positions. The program includes practical, 
problem-oriented instructional strategies that require students to work collaboratively in partner- 
ship with others, a curriculum grounded in the literature of effective practice, and a culminating 
project focused on major issues in educational leadership. 

Before fully implementing the program in September 2004, the faculty made some key deci- 
sions that influenced all that followed: 


e The faculty agreed upon the professional outcome focus of the program, which is the 
preparation of executive leaders; these are district-level leaders, such as superintendents, 
assistant superintendents, and directors of special education. That decision influenced 
every step of the design process. Building-level leadership is the focus of SLU’s master’s 
program, which was redesigned in 2006. 

e The differences between the Ph.D. program and the Ed.D. program were clarified and 
communicated to prospective students and other stakeholders so that the professional 
outcomes that students seek are matched to the program they enter. 

e The faculty linked Ed.D. program experiences to leadership practices. They aligned pro- 
gram elements and requirements that the students would encounter with the roles they 
would fill in their professional lives. Thus, the program itself was designed to provide 
numerous opportunities for students to apply new knowledge and skills in circumstances 
similar to the ones they would face as educational leaders. 

e Using the literature that grounded the work and provided a conceptual focus, the faculty 
identified four domains that would distinguish its Ed.D. program. The domains are (a) 
complex educational problems, (b) teamwork, (c) project management, and (d) school 
improvement and educational change. 


Theoretical Foundation and Program Components 


The use of the four domains and their interaction distinguish the SLU Ed.D. program from the 
vast majority of educational leadership preparation programs. The four domains are grounded in 
literature that is associated with effective executive leadership. Because empirical work has been 
somewhat limited in the area of executive-level leadership in school districts, SLU’s program 
developers included literature from various fields that could support program design decisions, 
including studies of adult learning, school leadership preparation, executive leadership, business 
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leadership, change leadership, organizational learning, and effective school-level leadership. A 
brief description of the four domains and their manifestation in the Ed.D. program follows. 


Complex Educational Problems. The most notable change in SLU’s Ed.D. program was 
the implementation of a culminating project that incorporates problem-based learning (PBL) and 
focuses on major issues in education. PBL is used in other disciplines, medicine and business, 
for example, because it enhances the critical thinking skills in graduates; however, it is not 
yet widely incorporated into educational leadership programs even though education graduates’ 
critical thinking skills are as important as those of doctors and business executives. A PBL project 
closely mirrors the kinds of experiences educational administrators are likely to encounter on the 
job. Leithwood and Steinbach (1995), in their study of superintendent problem-solving behaviors, 
noted that “On the whole, the results of this research indicate that senior school administrators 
appear to operate more like senior managers in other fields than like principals” (p. 91). 

The literature addressing pedagogy in leadership preparation suggests that problem-based 
learning is particularly useful, because it develops expertise in problem identification and problem 
management. In problem-based learning, students learn how to reflect both during a decision- 
making process and after decisions have been made (Schén, 1987). Additionally, Palmer and 
Major (2004) wrote that “PBL helps participants to develop a knowledge base. It teaches time 
management, collaboration, resourcefulness, research skills” (p. 121). 

In SLU’s Ed.D. program, students use PBL to work on team projects centered on current 
problems of practice. The project work occurs throughout the program rather than after the 
coursework is completed. The integration and application of coursework enriches the project, 
promotes program fluidity, and adds meaning and depth to the coursework. After engaging in 
the PBL process to investigate and articulate their educational problem, students identify one of 
three formats to complete their project. The format is chosen to match the problem they have 
articulated. Either the teams continue with PBL or they move to a policy analysis or product 
development format. Whatever approach they use, the roots of each are embedded in the PBL 
process that clarifies and focuses on complex educational problems. 


Teamwork. Practicing educational leaders do not work in isolation; they are a part of a 
system that requires interdependence and cooperation. Larson and LaFasto’s (1989) well-known 
study of the effective practices of work teams provides a foundation for SLU’s Ed.D. program. 
The authors define a team as “two or more people; it has a specific performance objective or 
recognizable goal to be attained; and coordination of activity among the members of the team is 
required for the attainment of the team goal or objective” (p. 19). This definition guides the team 
development processes that are taught and used during the 3 years of the Ed.D. program. 

Literature support for the value of teamwork is rich. For example, McREL’s research-based 
“Balanced Leadership Framework” identifies several effective leadership practices that occur in 
the context of teams. Educational leaders are members of teams when they “build capacity,” 
“develop a community of purpose,” and “create a condition of distributed leadership” (Waters, 
2004). 

Other scholars reinforced this team notion of effective leadership practice. For example, in her 
research synthesis of principal practices that are associated with student achievement, Kathleen 
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Cotton (2003) found that leaders in high-achieving schools are involved in outreach to parents 
and other community members; establish and maintain a collaborative culture; work with their 
staff members to share leadership and decisions; and engage staff members in professional 
development and collegial learning activities. Earlier work by Smith and Andrews (1989) offered 
similar findings of leadership at the building level. Their description of the instructional leadership 
practices requires collaboration between the leader and other stakeholders. 

More recently, Karen Seashore Louis and Kenneth Leithwood (2000) suggested that “We have 
increasing evidence that collective responsibility for student learning, in addition to improved 
technical teaching practices and curriculum, is a fundamental correlate of student achievement” 
(p. 283). Educational leaders must have knowledge and skills that give them the capacity to create 
cultures of collective responsibility. 

As further support, Spillane (2004) identified themes in educational leadership. His “distributed 
leadership” theme focused directly on the notion that when leadership is distributed it exists in 
“collaborative, collective and coordinated” distributions. This means teamwork. 

Finally, Vygotsky’s (Vygotsky, 1978, as cited in Henson & Eller, 1999) notion that students 
have a “zone of proximal development” that limits what they can learn on their own compared to 
what they learn from interactions with teachers and other learners also enhanced the theoretical 
framework that grounded SLU’s redesigned Ed.D. program. The faculty was convinced that the 
doctoral students would learn more and be able to apply what they learn more successfully if they 
learn in the social context of teams. 

Consequently, SLU’s Ed.D. students work in teams of three or four to complete their Ed.D. 
projects. Using PBL, each team of students identifies an area of interest associated with ed- 
ucational leadership, investigates the literature within the area, identifies the major educational 
issues associated with it, and crafts a problem statement that clarifies the students’ project goal and 
guides their project work. The team project format promotes professional learning, collaboration, 
and simulates the authentic experiences of educational leaders. 


Project Management. Understanding how to break down educational improvement pro- 
cesses into manageable projects is an essential and difficult leadership skill. This type of manage- 
ment is a key to effective leadership. In his important review of educational change, Matt Miles 
(1993) wrote that (effective leaders) “typically applied deeper, structurally oriented solutions to 
the more difficult, important problems, while being flexible enough to use a Band-Aid approach 
to minor issues” (p. 237). Thus, project management skills are essential. 

Studies of organizations suggest that project management “has become part of the core com- 
petency of organizations” (Cooke & Tate, 2005, p. 5). In fact, it is associated with change 
management in many organizations, including businesses, government, and education. Knowl- 
edge about project management has grown dramatically in the past two decades, providing a 
wealth of information about effective project management practices. For example, as described 
by Cooke and Tate (2005), the 2000 Guide to the Project Management Body of Knowledge 
provides definitions and standards that are used internationally. 

Project management is the application of knowledge, skills, tools, and techniques to project 
activities to meet project requirements. Project management is accomplished through the use of 
the processes such as: initiating, planning, executing, controlling, and closing. The project team 
manages the work of the projects, and the work typically involves: 
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¢ Competing demands for: scope, cost, risk, and quality. 
¢ Stakeholders with differing needs and expectation. 
e Identified requirements (cited in Cooke & Tate, 2005, p25): 


Ongoing, long-term school improvement is difficult to manage and monitor at the district level 
because of the complexity and interaction of change initiatives. Project management skills help the 
executive leader breakdown the ongoing comprehensive improvement process into manageable 
projects, which Lewis (2007) identified as a job that has performance, time, cost, and scope 
requirements and that is done only one time. 

In these shorter-term projects the phases of change are fewer and more precise. Cooke and 
Tate (2005) identified initiation, planning, execution, and closeout phases, whereas Taylor (2006) 
describes only three phases: development, implementation, and termination. 

In a study to determine the qualities of effective project managers, Posner (1987) organized his 
findings into six categories: communication skills (listening, persuading), organizational skills 
(e.g., planning, goal setting, analyzing), team-building skills (e.g., empathy, motivation, esprit 
de corps, creativity), leadership skills (e.g., sets example, energetic, vision, delegates, positive 
attitude), coping skills (e.g., flexibility, creativity, patience, persistence), and technological skills 
(e.g., experience and project knowledge). With the exception of the team-building and technical 
skills related to project management, these skills are embedded in most educational leader- 
ship programs. Intentionally and explicitly addressing the areas of team-building and project 
management enhances the SLU graduates’ abilities to manage educational change and school 
improvement more effectively. 

Project management is as integral to the success of the Ed.D. team projects as it is to the success 
of projects that district leaders manage every day. Yet, unlike business leadership preparation 
programs, project management is rarely included in the curriculum in educational leadership 
preparation programs. Working in teams on the Ed.D. projects affords SLU’s students a project 
management apprenticeship during which they practice and cultivate skills to manage projects 
effectively while receiving continuous feedback from faculty and the other team members. 


School Improvement and Educational Change. SLU’s faculty holds the assumptions 
that effective executive leadership in education is leadership for improvement and that school 
improvement occurs only when improved teaching and learning are the outcomes. For years, 
theorists and researchers have built a body of knowledge regarding the improvement of education 
(for example, Crandall et al., 1982; Creemers & Reynolds, 1989; Cuban, 1990; Datnow, 2006; 
Fullan, 2001; D. Levine, 2004; Louis & Miles, 1990). 

Embedded in that work are concepts of organizational learning, school reform, educational 
change, and so on. Whatever the term, the process of improvement does not exist in a vacuum. The 
organizational context or the environment in which executive leaders must move their systems 
forward is quite influential to the outcomes. Hoy and Sweetland (2001), in their discussion of 
school structures that are enabling or coercive to school development, reviewed literature that 
described influential attributes of school contexts. Others, like Bolman and Deal (2008), described 
four foundational frames that make up organizations in general and then focused their discussion 
of leadership on the improvement of the organization using the frames. 
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Though the context is powerful, the role of executive leader as a learner is also notable. Bennis 
and Nanus (1985) suggested that the quality of fostering organizational learning by example may 
be one of the most important functions of leadership. 


If the leader is seen as an effective learner from the environment, others will emulate that model, 
much as the child emulates a parent or a student emulates a teacher. The leader and the organization 
nurture each other, guiding the process of creative self-discovery by which each learns how to be 
most effective in a complex and changing environment. (p. 205) 


Additionally, when Burt Nanus (1992) described the complex, changing world in which 
organizations must survive, he argued for a flexible, adaptable, innovative-learning approach 
that is guided by the leaders in such a system. Manz and Sims (1989) made a similar argument 
regarding organizational effectiveness when they presented their case for “superleadership.” 
Kotter (1996), Glasser (1994), O’ Toole (1995), and others follow the same line of reasoning. 

In short, effective executive leaders are essential to sustaining effective, continuous educational 
improvement within the complex organization or context in which they work. In his study of six 
high-performance school districts that served significant numbers of low-income or minority 
students, Cawalti (2003) wrote that, “A key finding of this study was that central office leaders 
have shifted their focus toward improving those processes that contribute to higher achievement 
rather than focusing on more traditional roles of control, resource allocation, and supervision” (p. 
3). Additional support for this kind of executive leadership in school districts comes from a report 
entitled, Superintendents for Texas School Districts: Solving the Crisis in Executive Leadership. 
The author, John Hoyle (2002), reported that, 


Top-down bureaucratic management is being replaced by bottom-up executive leadership that leads 
by example, is driven by a vision of success, and encourages shared decision-making. These new 
superintendents must be bold, creative, energetic leaders, who can respond quickly to issues ranging 
from dealing with social change, information technologies, diverse student populations, demands for 
equity, and continuous school improvement for every child. (p. 15) 


In summary, this literature suggests that the nature of organizations requires that an ef- 
fective leader adapt to new demands within the organization while managing and sustaining 
improvements. The effective district leader is driven by the needs of the learners; faces complex 
systemic-wide issues; and provides structure, guidance, and support for the organization to move 
through the phases of continuous improvement. To that end, SLU’s four foundational domains 
‘nteract and reinforce each other. That is to say, effective school district leaders address complex 
problems, support team work, and implement project management structures to improve schools 
for students. 


PROGRAM DESIGN AND COMPONENTS 


One unifying theme that connects coursework and the team projects in SLU’s Ed.D. program is the 
expectation that graduates will facilitate school improvement. All students study organizational 
context and leadership in their first semester research topics course, where Bolman and Deal’s 
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(2008) Reframing Organizations is a text. They also study, apply, and assess the principles of 
team work, supported by Larson and LaFasto’s (1989), TeamWork and project management, 
supported by Lewis’s (2007) The Fundamentals of Project Management. Courses from finance to 
politics to school-community relations are taught with an emphasis on making improved student 
learning the central concern of educational leadership. Finally, specific knowledge and skills 
regarding the phases of school improvement and educational change are embedded in courses on 
curriculum leadership, professional development, human resources, and, most importantly, in the 
Ed.D. culmination projects. 


Key Features of Projects 


Key features of those culmination projects are described in an Ed.D. Program, Student Handbook 
(Saint Louis University, Department of Educational Leadership and Higher Education, 2007). 
They include the following: 


Students are grouped into teams of three or four to work on their projects. 

e An approved faculty advisor is assigned to each team of students when the team is formed. The 
team members ask a second faculty member to become their reader after they have identified 
their critical problem. 

e Ed.D. students complete a project and a project report instead of a dissertation. Each project 
begins with the process of problem-based learning. Through a review of the literature, students 
clarify their educational issue and prepare a problem statement. They select one of the following 
three approaches to complete their project: policy analysis, problem-based learning, or product 
development. 

¢ Whereas a written comprehensive examination is based on the program’s curriculum, the oral 
examination focuses on the project. Each student must pass an oral examination that addresses 
the team’s project work and the individual’s analysis of the project work. 


Program and Project Phases and Timeline 


In order for students to complete the program in 3 years, as shown on the timeline (see Table 1), 
they must meet all milestones at the times indicated on the timeline (Saint Louis University, 
Department of Educational Leadership & Higher Education, 2007). Students graduate when they 
complete all the requirements of the program and satisfy the department and graduate school 
review of those requirements. 

The Ed.D. project requirement is designed to enhance students’ problem-solving skills, com- 
munication skills, and content knowledge so that they, as educational leaders, can improve educa- 
tional practices. The four foundational dimensions are embedded in the project requirements. For 
example, one of the first tasks that project teams complete is the development of team agreements 
for working together. One essential characteristic of effective work teams is that they discuss 
how they will work together. The guidelines include, but are not limited to, decision-making, 
communication, participation, management/facilitation, problem and conflict management, and 
confidentiality. When the team members have written their guidelines, they sign and date them. 
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Once the teams are formed and the guidelines are written, the team members develop a 
preliminary project management plan to guide their work until they define the project problem 
on which they will work. The identification and clarification of the project’s complex problem 
requires the team members to read the literature associated with the topic they have selected. 
On average, the teams will read for 4—6 months before they are able to identify and clarify their 
project’s problem. 

Over time, the preliminary project management plan is revised to become a full project 
management plan. The use of project management plans to guide the work of the teams is 
essential in that the plan focuses the tasks, guides decisions, and supports the team in completing 
the work. Once a project team has been formed, the team agreements have been written and 
signed, and the preliminary plan is completed, the problem-based learning process is initiated. 
Clarification of the problem allows each team to select the approach they will use to address the 
problem and complete its Ed.D. project. Students select one of three approaches: problem-based 
learning, product development, or policy analysis. Three faculty committees were responsible for 
the development of the approaches. Each used the literature in their area to guide the work. 

Problem-based learning is a self-directed group activity that primarily uses constructivist 
learning strategies. Students work in teams as collaborators, problem-solvers, designers, and 
information gatherers to create artifacts that represent their learning. Essentially the students 
are the workers and the advisor serves as a facilitator. The team members’ work consists of 
continuously responding to three questions: What do we know now about the issue? What do we 
need to know? And what are some possible actions/solutions we might use to address the issue? 
Each phase of the work introduces situations for which students are likely to be unprepared to 
address. Thus, new questions emerge and are addressed in a cyclical process of investigation. 

Policy analysis is a process through which team members investigate the cultural, economic, 
historical, legal, political, and/or social dimensions of a policy affecting the quality of practices in 
elementary, secondary, and/or higher education. The project teams use a systematic, step-by-step 
process to analyze the nature, rationale, impact, effectiveness, and/or implications of existing 
or alternative policies. In all such work, the analysis is based on evidence-based standards or 
normative criteria that the students identify by reading the literature. 

Product development is a research and development process that is used to create a “product” 
that is needed to address a critical problem in the field of education. The product development 
process involves identifying a complex educational problem and investigating the literature and 
current practices associated with that problem to articulate an educational need. The product 
development steps include: needs assessment, product planning and design, proof of concept 
analysis, product creation, preliminary field-testing, product revision, final product testing, and 
final product revision. 


Team Reports, Individual Reports, and the Examination 
To graduate, project team members complete two reports: The Team Project Report and the 


Individual Analysis Report. 


Team Reports. The Team Project Report presents information about the topic, the educa- 
tional problem that the team addressed, the literature that informed the project, the strategies that 
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the team used, the project outcomes/findings/analysis, and recommendations. Project Reports 
are published through Dissertation Abstracts, which means that, once they are approved by the 
department faculty, they are reviewed by the doctoral advisor in the graduate school, revised by 
the students, and approved by the graduate school dean before graduation. The reports vary in 
format to match the approach that the team used to address the problem. 


Individual Reports. Each student also completes an Individual Analysis Report. That report 
provides a focus for the student’s oral examination. The policy of the graduate school and the 
department is that an individual receives an Ed.D. degree when he or she has completed the 
requirements of the program. Teams do not receive Ed.D. degrees. Consequently, the Individual 
Analysis Report is extremely important. It is a comprehensive and substantive analysis that must 
meet the rigorous requirements of a doctoral degree. Each student must write his or her Individual 
Analysis Report without input from or consultation with other team members. 


Examination. The oral examination is the final activity for the Ed.D. team project; it ad- 
dresses the Team Project Reports and the Individual Analysis Reports. All team members par- 
ticipate in their oral examinations on the same day. The team’s advisor, the reader, and three 
additional faculty members make up the committee that administers the oral examination. At the 
beginning of the examination, the project team and the five committee members address the team 
project together. This is followed by the examination of each team member, one at a time. The 
content of that oral examination focuses on the student’s individual analysis report as well as on 
his or her mastery of the required readings and knowledge of the field of educational leadership. 
Each individual examination takes approximately one hour. 

Occasionally, some team members are ready to participate in the oral examination, and others 
are not. In that case, the advisor and the reader will arrange for the oral examination of the 
students who are ready and will continue to work with the students who have more to do before 
they can take their oral examinations. 


Assessment 


Student Assessment. The faculty assesses students’ work based on the expectations that 
are associated with doctoral-level achievements. The evaluation criteria used in SLU’s Ed.D. 
program are based on the work of Ernest Boyer (1995) at the Carnegie Foundation. Boyer 
identified six assessment areas: clear goals, adequate preparation, appropriate methods, significant 
results, effective presentation, and reflective critique. Specific indicators and a measurement rubric 
for each area are used to give feedback to SLU students. 

The oral examination is given by the advisor, a reader, and three faculty committee members, 
using graduate school ballots to record their votes. Final graduation ballots are signed by the 
student’s advisor and the reader once the student has satisfied all the requirements for graduation. 
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Program Assessment. In addition to the less formal assessments of the Ed.D. programs 
that occur each year, including student evaluations, faculty reviews, and program development 
committee revisions that are based on the information regarding the program collected from 
current students, the department has a more formal evaluation process that assesses its graduates 
annually. 

That evaluation is based on data collected through questionnaires and interviews with graduates 
as well as with the people for whom they work, in years 1, 3, and 5 after graduation. Since 
students began graduating from the new Ed.D. program in spring of 2006, the evaluation design 
was enhanced to address specific areas of interest to the faculty regarding the practices of 
program graduates. For example, SLU’s interest in educational improvement is highlighted in the 
evaluation. The 2008 report was just disseminated to the faculty at its annual faculty retreat and 
will be used to guide changes to enhance the Ed.D. program. 


CONCLUSION 


Engaging in its own problem-based learning, the SLU educational leadership faculty studied and 
planned for the better part of 2 years to design the new professional doctorate program. The 
faculty was aware from the beginning that the new program had to pass the approval of the 
graduate school at a university that valued its standing as a research institution. Even with careful 
preparation, the jump from the old program to the new one, in September 2004, seemed dramatic. 
Four years later, though the success of the initial design has been extraordinarily satisfying, the 
faculty’s capacity to reflect, learn, and adjust has enriched the program and stabilized its future. 
Some of the “lessons learned” are worth sharing. 


e Since the university graduates individuals, not teams, designing a process in which high- 
quality, individual doctoral-level products accompany the culminating team product was es- 
sential. However, until the first teams moved through the program, none of the faculty members 
knew for sure that the design for that requirement would allow them to differentiate quality 
among students on each team. To that end, the final products, which are a combination of a 
team project report and the team members’ individual reports, have been amazingly effective. 

¢ Requiring the teams to develop group norms, comprehensive problem statements, and project 
management plans has been important. Initially, the faculty taught about these procedures 
but did not require teams to use them in the project team, assuming that the project teams 
would accept the importance of the tasks and would complete them. Over time, it became 
clear that successful project teams discussed and worked on these tasks, whereas teams that 
struggled did not. In other words, the project teams did not automatically translate the theories 
of teamwork, problem identification, and project management into action. Requiring that they 
complete the tasks was essential to team success. 

e The flexibility that was built into the formats for team project reports and team products 
was more valuable than anyone anticipated. The inevitable move toward the five-chapter 
dissertation format began to creep back into the program early on because it was what people 
recognized as doctoral work. Working with students to design reports that were aligned with 
the topic, the approach, and outcomes of each teams’ project blocked that creep effectively. 


98 S. TOFT EVERSON 


e Producing a student handbook has been a major step i clarifying the process for students. 
Individual handouts and advisor suggestions were not as comprehensive and integrated as they 
needed to be. Now, the advisors have a resource that supports the students when they face the 
rigor, ambiguity, and complexity that is a part of all doctoral work. Additionally, examples 
of completed projects are now available for students to review. The quality of these products 
has eliminated the rumors about how much easier a team project would be than an individual 
dissertation. 


The continuous development of SLU’s Ed.D. program has become routine for the educational 
leadership faculty. Faculty retreats, the program development committee, department leaders, and 
students stimulate questions, reflection, and revisions. The faculty’s commitment to continuous 
improvement in its own program provides a model to the students that emphasizes that leadership 
for improvement is foundational to the SLU program. The next step for the department’s program 
development will focus on its Ph.D. program. The goal will be to redesign the Ph.D. program with 
the same commitment to excellence and substantive student outcomes that guided the faculty’s 
development of its professional doctorate. 
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A nationwide strategy for improving Ed.D. programs is needed to overcome two important dilemmas 
that are typical in schools of education: low academic status for Ed.D. programs and an overre- 
liance on tuition as a source of revenue. Other major hindrances are the lack of agreement about 
research-based standards, a weak alignment with other elements of educational reform, and fuzzy 
accreditation standards. The authors then discuss two broad strategies for reversing the downward 
spiral in educational leadership programs: a compact among elite schools of education to improve 
their Ed.D. programs and a National Academy of Educational Leadership. For both strategies, the 
authors explore lessons from a range of analogies drawn frome experience both in the United States 
and internationally. 


Economic and political dynamics propelling programs that presently proffer the Ed.D. simply 
do not permit the prospect of widespread institutional self-improvement. The majority of public 
and private professional degree-offering universities are deeply status deprived and intensely 
tuition dependent. Consequently, they cannot risk elevating Ed.D. admission and graduation 
standards for fear of jeopardizing enrollments and attendant revenue streams. To strive to improve 
is to risk decline. 

Under current arrangements, the field is paralyzed. It is difficult for even the most visionary 
of colleges to withstand the revenue peril and reform its Ed.D. To do so jeopardizes its applicant 
pool and assuredly shifts tuition-paying enrollees to even less selective institutions. If there is to 
be a change in the character, rigor, and regard of the Ed.D, the impetus likely will have to emanate 
from outside sources. Reform will have to be collective in nature. Either all have to be at risk, or 
none can afford a risk. 

There are at least two viable means by which external influence could come to bear. One 
strategy involves voluntary formation of a self-regulating association by an elite group of univer- 
sities pledging to establish and adhere to high admission, program, and performance standards. 
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The second strategy necessitates formation of a National Academy of Educational Leadership, 
federally chartered and financially subsidized, and driven to high standards by a national board 
of prestigious laypersons, academics, and educational practitioners. 


STRUCTURAL IMPEDIMENTS TO SELF-IMPROVEMENT 


Doctor of education (Ed.D.) program deficiencies are deeper than the immediate details of the 
degree programs themselves. The problems are endemic throughout virtually all the components 
of United States education leadership preparation. Mechanisms by which the United States 
presently recruits and prepares its school leaders are so fragmented, unsuited, and haphazard 
as to provide little assurance that those selected to lead schools can fulfill the nation’s elevated 
educational aspirations. 

This criticism is not directed at specific individuals or categories of education leaders. Through- 
out the nation, there have long been and continue to be sterling examples of principals, superin- 
tendents, and state and federal officials whose creativity and courage have already made a huge 
difference for students under their authority. Rather, this lament pertains to the crazy quilt manner 
by which leadership is identified and nurtured. It is the entire system that is in need of repair. 
Interestingly, however, reform of the Ed.D. offers a productive point at which to begin to alter the 
larger leadership preparation system. 

The United States has approximately 100,000 elementary and secondary schools nested in 
14,000 public school districts. Each of these agencies, within the boundaries of any particular 
state’s broad professional certification requirements, establishes expectations for the employ and 
performance of its school leaders. 

However, the problem is not so much decentralized selection, though such may be a con- 
tributing element. After all, there are also thousands of hospitals and churches and millions of 
businesses, each selecting a leader. These latter endeavors have known and longstanding standards 
by which leadership candidates can be appraised. The problem with public schools is the absence 
of effective centripetal forces shaping the leadership expectations and preparation standards for 
education’s thousands of decision-making units. 

To be sure, minimum state leadership standards exist, but a candidate’s compliance with 
them is measured more by a transcript of course-taking than any test of performance capacity. 
Also, national and professional organizations episodically issue admonitions regarding desirable 
leadership characteristics and training. Seldom are these anchored in research and, even when 
they are, they have no regulatory weight. 

The problem of state leadership standards typically has three dimensions. First, almost every 
state has a statewide set of standards for what students are expected to learn. But these state 
standards often focus exclusively on academic disciplines alone when the learning demands of 
the 21st century require complex problem-solving, new forms of literacy and communication, and 
abilities to invent and work in groups. In short, there is the dilemma of tying a transformational 
Ed.D. leadership program to what is often seen as an interim view of powerful student learning. 

Second, most states then have leadership standards for their administrator credential programs 
that are aligned with the standards for what students are expected to learn, but this alignment is 
more bureaucratic than conceptual or research based. Administrator credential programs often 
have regulatory weight but little substance that will transform student performance by national 
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or international perspectives. In turn, Ed.D. programs in those same states often have very little 
alignment to state standards but often have more conceptual and research-based perspectives on 
learning. 

Finally, there is a problem with the alignment of leadership selection, training, and job ex- 
pectations after the training. Tucker and Codding (2002) describe the advantages of this form of 
alignment in military training where only some junior officers are invited to join the extensive 
training program for the next tier of responsibility. Once that training is completed, the job perfor- 
mance expectations at the new military rank are aligned with the training, as is the performance 
assessment system. 

In contrast, the pipeline supplying America’s education leadership candidates is comprised 
of literally thousands of disconnected and competing schools of education located throughout 
the nation. The overwhelming majority of these are heavily dependent upon enrollee tuition for 
their economic survival. Moreover, they are intensely status deprived and seldom have sufficient 
self-esteem to act courageously. 

Unlike America’s nation’s military academies and preeminent professional schools, these 
status-deprived and enrollment-dependent education schools can ill afford to elevate admission 
standards or to require rigorous study on the part of their admitted students. Hence, for potential 
enrollees, to apply is to gain admission and to be admitted is to graduate. The coursework 
involved is typically offered on tired time, after the school day is over, or on weekends. It almost 
always is taken in conjunction with full-time employment. Virtually none of the programs require 
residential study. This is in sharp contrast to the professional preparation expected of physicians, 
attorneys, ministers, engineers, and high-level business executives. 

The Carnegie Foundation for the Advancement of Teaching, under the leadership of Lee Shul- 
man, has been studying professional doctorates across six professions: medicine, law, nursing, 
the clergy, engineering, and education to learn more about the core design of these programs. A 
multiyear study has produced findings about individual professional doctorates in other profes- 
sions and an initiative to work with a set of approximately 25 schools of education to create a 
network of action around design principles drawn from this study (The Carnegie Project on the 
Education Doctorate, n.d.). Shulman, Golde, Conklin Bueschel, and Garabedian (2006) have also 
made an important distinction between a Ph.D. and an Ed.D. degree in ways that are crucial to 
the advancement of exemplary professional doctorates in education. 

Some education leadership preparation institutions are accredited. However, accreditation in 
this setting carries nowhere near the consequences faced by medical, law, engineering, and other 
professional schools. Rather, in educational administration generally and for Ed.D. programs 
particularly, ad hoc accreditation teams often consist of a conflicted and self-serving tripartite set 
of participants including faculty from similar training institutions, graduates of the same places, 
and cooperating state officials. The modus operandi is go along to get along. 

The consequence of this haphazard condition is an annual outpouring of tens of thousands of 
individual administration graduates, including thousands of Ed.D. holders, for whom little can be 
guaranteed regarding their general knowledge and leadership capacity. It is difficult to impossible 
for a visionary university Ed.D. preparation program to break away from this low-profile pack and 
initiate higher standards for its admission and performance expectations. To do so is to place itself 
at financial risk because its potential pool of applicants and enrollees may well shy away from 
the aspiring institution and simply flow to another, competitive, university with lower standards. 
After all, the direct and transaction costs likely will be the same, and the larger profession and 
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general public probably will not be able to differentiate a graduate from any particular institution 
from that of another. 

Moreover, geographic accessibility is sometimes, but not often, a challenge. The substantial 
20th-century expansion of directional universities, East or South this, North or West that, and 
Central something, virtually guarantees a publicly subsidized Ed.D. program within a 2-hour 
drive. Failing that, there are hundreds of private institutions that can fill the gap. Their tuition will 
assuredly be higher, but their facilitation of federal student loan acquisition is probably greater. 
And, for those for whom geography is a genuine challenge, there exist online programs wherein 
the Internet removes virtually all geographic barriers to access. 

These structural conditions create serious constraints for both the prospective student and 
the Ed.D. program at a university. For the prospective student, there is little incentive to seek a 
high-quality degree program—which is quite different from wanting a diploma from a prestigious 
university. Employers don’t make much differential among job candidates based on the quality 
of the Ed.D. program itself, and there are few incentives in the job itself for having completed 
a high-quality program. In medicine, by contrast, surgeons can lose privileges at a hospital if 
their work is of low quality and in the law, an attorney can be sued if his work is substandard. 
In education, an Ed.D. program graduate finds a lack of alignment with other aspects of the 
education system and little incentive to be accountably excellent for their leadership. 

The faculty in a specific school of education faces major hurdles if they aspire to dramatically 
improving their Ed.D. program. With little guidance from research, accrediting agencies, or 
external assessments of graduates across Ed.D. programs, leadership teams have little guidance 
as to what would count as quality. The ratings by U.S. News exacerbate this problem. In education, 
no distinction is made in the criteria that define quality in Ph.D. and Ed.D. doctoral students, so 
a specific university makes this differentiation at its own peril. 

This peril of external criticism, and the competition for students among universities, creates 
several internal dynamics as well. First, the conversation about the design and direction of the 
new Ed.D. program stays too much within the tenure-track faculty. Many schools of education are 
literally prevented by faculty from even considering a new direction because of faculty interests 
in maintaining existing faculty load arrangements and/or in protecting what little status has been 
achieved within the school of education. At the same time, practice leaders from outside of 
academe have rarely been brought into sustained and important conversations about program 
directions. 

Second, there are few fiscal resources to invest in creating a high-quality Ed.D. program. So, the 
“tired time” problem of part-time doctoral students engaging in after school and weekend study 
is mirrored in “tired time” program development for willing faculty. At best, new Ed.D. programs 
will make vibrant use of video, case studies, distance-learning components, and performance 
assessment in real-world settings. These elements are exactly the most time-consuming and 
expensive to produce in high-quality Ed.D. programs. 


REVERSING THE DOWNWARD SPIRAL 


External forces are needed to reverse this widespread mediocrity. Two strategies have a prospect 
of provoking reform. One is nongovernmental and voluntary. The other is possibly governmental 
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or philanthropic in nature but at a minimum will require central action. These are not mutually 
exclusive activities and would be reciprocally reinforcing, if taken together. 


An Elite Compact 


There are a few, a very few, elite universities that have selective admissions for their Ed.D. 
programs. This is not to proclaim that their programs are exemplary or that their entry standards 
are sufficiently rigorous. The only claim here is that the larger institutions in which such programs 
are embedded possess sufficient status and financial heft that there may exist the prospect of a 
willingness to engage in risks. 

These selected institutions, as many as but probably no more than 10 of them, need to form an 
exclusive association pledged to the development and adoption of elevated Ed.D. standards. Such 
a consortium for educational leadership excellence must act collectively to advertize its members’ 
new and higher standards and thereafter police itself to ensure that all components adhere to the 
agreed-upon criteria for admission and candidate program completion. The admission standards 
and curriculum components would then have the prospect of defining and elevating the field. 

The 10 members of the consortium for educational leadership excellence might agree to 
share courses, rotate faculty, and permit participants to enroll in courses among them. In time, 
other institutions could apply to be a part of the consortium. Those who met announced criteria 
for participation, and who pledged themselves to adherence to consortium standards, could be 
admitted to membership. The long-run escalation of standards among this initial elite might come 
to characterize the field. 

The Carnegie Initiative on the Professional Education Doctorate (CPED) is one proxy for 
the elite compact strategy (The Carnegie Project on the Education Doctorate, n.d.). CPED has 
been able to assemble a collection of approximately 25 universities in a network of support and 
collective action to improve their Ed.D. programs and has been able to draw on the extensive 
design features found in the Carnegie study of the professional doctorate. With support from the 
Spenser Foundation, CPED is working to establish a “low stakes and high payoff” effort featuring 
a few common standards and related assessments that institutions would engage in on a voluntary 
basis. It remains to be seen whether CPED can engage a sufficient number of elite universities, 
create a powerful change process within each institution and across institutions, counter the force 
of incentives and external ratings of Ed.D. programs, or provide sufficient time and resources for 
exemplary and comprehensive Ed.D. program reform. 

In any case, the elite compact strategy is not a rapid solution to today’s problems. It might well 
take a quarter of a century for such self-improvement dynamics to have an appreciable effect. 
Still, what is clear is that if left to pursue their present day path, Ed.D. granting institutions are 
not likely to improve. 


A National Academy of Educational Leadership 


Legislative authorization, by the U.S. Congress, or formation by a prestigious foundation or 
consortium of foundations, of a National Academy of Education Leadership might be a productive 
vehicle for elevating Ed.D. degree standards. Such an agency or body would have as is guiding 
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and policy-setting body a Board of Educational Leadership Excellence comprised, for example, 
of prestigious and visible private sector business leaders, academics, practicing school leaders, 
and relevant researchers. This policy body would construct leadership preparation standards 
and degree criteria. Thereafter, this body either would form an operating National Education 
Leadership Graduate School, an analog to a Military War College, or frame bids by which a 
consortium of universities and other institutions might compete to receive financial support in 
exchange for serving national education leadership purposes. 

The training institution, whether government or foundation supported, or both, would be 
responsible for the admission and competitive selection and professional preparation of a cadre 
of capable and highly visible leaders from which the nations districts and schools could select 
candidates and would be authorized to conduct and financially support rigorous research regarding 
leadership. The lighthouse nature of a national academy of educational leadership would serve 
to illuminate high standards against which graduates of other programs would be compared. It is 
through this modeling and comparison that Ed.D. program quality might, eventually, be ratcheted 
upward. 

There are important examples of what such a National Academy of Educational Leadership 
could look like and, in turn, some implications and lessons for its design, governance, and 
networking with schools of education. As mentioned above, the Military War College is a strong 
example of linking selection of qualified and experienced students to problem-based leadership 
issues and subsequent job expectations. This example also shows what can be done with ample 
funding and important partnerships across military services and related national security agencies. 

In the U.K., the National College of School Leadership (NCSL) provides extensive training 
and assessment for leaders across the spectrum from those school-based people considering a 
career in organizational management, to entry-level selection and training, to ongoing professional 
development for newly appointed and veteran school heads, to those who will be coaching school 
heads. The national training facility in Nottingham features a hotel and extensive training facility 
and has links to regional partners across England. But NCSL has not adequately linked to schools 
of education, which are often portrayed as “the problem,” or enhanced emerging Ed.D. programs 
in the U.K. Moreover, some have questioned the intellectual and research base of NCSL and 
university professor leaders have often turned to the School Trust as a better source of idea 
sharing and professional development for cutting edge schooling. 

In the United States, there are important lessons to be learned from non-university providers 
of educational leadership development such as the Eli Broad Foundation’s Superintendents 
Academy, the National Institute for School Leadership, and New Leaders for New Schools. 
Each of these U.S. examples shows what extensive funding, careful selection of candidates, ex- 
tensive program development, and a unique niche in linking to school systems can accomplish. 
There are important lessons for Ed.D. programs in these examples. 

There are also examples of national networks to assess the competence of Ed.D. graduates. The 
National Board for Professional Teaching Standards is working to assess administrative leaders, 
after years of work on assessing and certifying exemplary teachers. In other professional doctorate 
programs, state law board exams provide the basis of licensing attorneys and have working 
relationships with law schools. A better example might be assessments of prospective dentists 
in that the exams balance content knowledge, clinical knowledge, and clinical performance 
assessment. The assessments are also closely linked to the dental school curriculum, provide a 
problem-based research-utilization approach to dentistry that can be continued by graduates in 
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their practice, and happen throughout the years of dental school for a student. These approaches 
to assessing the competence of candidates also have important lessons for Ed.D. programs. 

In summary, National Academy of Educational Leadership could cooperate with or stand 
separate from the previously described elite compact. In any case, it is clear that a major investment 
in quality program delivery linked to a robust performance assessment system will be needed to 
help schools of education mount significant reform in their Ed.D. programs. In addition, there will 
need to be a national strategy for defining and accrediting quality Ed.D. programs and providing 
appropriate ratings of excellence so that status issues can be addressed. 


REFERENCES 


The Carnegie Project on the Education Doctorate. (n.d.). Retrieved October 6, 2008 from The Carnegie Project on the 
Education Doctorate website, http://cpedinitiative.org 

Shulman, L., Golde, C., Conklin Bueschel, A., & Garabedian, K. (2006). Reclaiming education’s doctorates: A critique 
and a proposal. Educational Researcher, 35(3), 25-32. 

Tucker, M., & Codding, J. (Eds.). (2002). The principal challenge: Leading and managing schools in an era of account- 
ability. San Francisco: Jossey-Bass. 




















« Lee TT) 
p gordo @ her : a": 
ev 4 a vl & ai p> ah 
iy cove ae a 
Riart 3 ant ee 2h ee 3 — eon 
s me: wa parr - 
Tt Fao fat Foye + oped tm @ a sesh < a gil 
Mer REND csr /tiNagle Me any = 
Pes rh (Veron: peau J ; “ante a 
ehh ae r AZ bea Be © no i Aes 
loa aerate: (hie Cation a A r Racodh® Xa 
Jes lang ve aon (breve, Usage Tm aaa ah 
, - te tye eae an hing wig rae, * & AS ig 


as 
abs 
ae 
Fi = 
















rile, (peanh wel laren «6 6 ee» 
, . oe Pee tod Ql ow * ea) p-7 7 
pe ot Hiv. cee ~~ av FAs fF wo ¥ wr 
me rigs hag ph. agp “See w 













































i, slate ne CAes i wee sa 
Setel csiohers Sianiteeiaer £ Map ieee 

peeve cham. heme 7 - 
mr Yaeger A. ge) Poe Se ; ae f 

i Sequd of KES nr) MY 0 lad 

V Wehner rad mn ett. Sen was 


tee ong até ru i liny y 


ac ty Plime freer pats | > 










Awnrins es esc ceees - | eats, it 
7 a 

im try se) ee »< : o a - Wf = at r 

peed? Nil an aed hy ay ¥ ¥ 





Kan Fieg'6. Sley = eegae. a 
i ed epi (<2 oe 2 : : 
iPbdlaaty Jems id ake , i a) es 

Al ’ wed) Mie ar (fer Ww prea’ * , apa mie 
hh io veh Vika a 1 ; ar “am “4 al 
Pes | Righe! Lis wer 09 ve ‘ e : ar 
> TV days cr angie j 

Sieh ear 









‘Tere w Da _ 
pt Pas \% sq | = <u & 
Pikes Repparda inn, CU) 8S. oe oS eer 
} weiter an). apertory ny go ime . i 
ui, Choke: caviar is onl 
Se ee ee ad : < 
pie BSTS opp Hr »* dalagit my. i ; f 
08 ture ehed for ehh va ’ 7 ee ; 
Vigem Palohed at ees ; 
ee 
eich hed, Dy re <p 









100 LCRA eae 7 
















“Veet ew eu titt, anttl: aioe a yektioge ior he ieined 
fi Geescdew he Conmetonce: eh nant ry Naat 
by wie rhea hiatal Aedoony ait tbteetantat \ankhorn 
nartAte aves ithe prewionisty rite nar ei OMAR tie aycan, 
io quality pragrars. delivery hdhiysl i ® lank peter thane 8 et 
rele wohovihs of cafvonaign mieecl higrtey see Pehen eetenty SD ‘ a 
Aeed try bt A jenni suastegry Seen ieee macwoeliy te, Spang 
pee Aime hat, vata fl mllances ‘i ome an Sa ems a set 


- We 
ReeeMNCES 
an ic 

Tia Cawee Cy wee oe sien - pi ip Arh Fhe Ma 
| keith Ram ae 9h he oe ne aaa). 
eS a ‘aie A ve et heuleon # q Sis, \ 
sige Me nt hate PRD 7 

eee a ee ae Ce gona epee» flies aw: in 
5 eh Te a Gam Dere-be- he a 


he 


i We % 
wer 


INSTRUCTIONS FOR AUTHORS 


The Peabody Journal of Education: Issues of Leadership, Policy, and Organizations is the nation’s second oldest scholarly publication 
concentrating upon education issues, particularly education policy issues. 
Content: The Peabody Journal of Education does not typically accept or commit itself to review individually submitted manuscripts. 
Rather, the Peabody Journal of Education: Issues of Leadership, Policy. and Organizations (PJE) publishes quarterly symposia in broad 
areas of education, including but not limited to topics related to formal institutions serving students in early childhood, preschool, 
primary, elementary, intermediate, secondary, postsecondary, and tertiary education. The scope of the PJE includes special kinds of 
educational institutions, such as those providing vocational training or the schooling for students with disabilities. PJE also covers 
institutions, and education matters that are important to nations outside the United States. Finally, it includes topics that are linked 
to the social and organizational context in which formal and informal education take place. 
The Editor cooperates with groups of scholars to present multifaceted, integrated expositions of important topics. A given issue of 
PJE may contain contributions from social scientists, historians, philosophers, attorneys, practitioners, and policymakers. Unsolicited 
proposals for special issues — including designation of participating scholars and an outline of articles — will be accepted for review. 
Additionally, the Editor cooperates with Editorial Board members to identify potential topics, guest editors, and contributors. PJE 
has the flexibility to consider publishing monographs or a series focused on particular lines of inquiry. In all cases, the Editor and 
the Editorial Board will ensure that each issue is carefully reviewed and its articles will comprise a high-quality contribution to 
understanding and practice. 
‘To submit a prospectus for a special issue, note submission requirements presented below and contact the Associate Editor, Meredith 
Ross (meredith.a.ross@vanderbilt.edu), for complete details. 
Manuscript Preparation: Prepare manuscripts according to the Publication Manual of the American Psychological Association (Sth 
ed., 2001; available from APA Order Department, P.O. Box 2710, Hyattsville, MD 20874), especially with regard to reference lists 
and text citations. Follow Guidelines to Reduce Bias in Language (APA Manual, pp. 61-87). Manuscripts should include the theme 
title and rationale for the theme. Using 1-in. margins, type all components double spaced and in the following order: title page (p. 
1), abstract (p. 2), text (including quotations), references, appendices, footnotes, tables, and figure captions. On page 1, type article 
title, author name(s) and affiliation(s), running head (abbreviated title, no more than 45 characters and spaces), author notes and 
acknowledgments, and name and address of person to whom requests for reprints should be addressed; on page 2, type an abstract 
of no more than 150 to 200 words. Indent all paragraphs. Use footnotes sparingly. Attach photocopies of all figures. Number all 
manuscript pages (including photocopies of figures). 
Cover Letter, Permissions, Credit Lines: In a cover letter, include the contact author's complete mailing address, e-mail address, and 
telephone and fax numbers. State that the manuscript includes only original material that has not been published and that is not under 
review for publication elsewhere. 
Authors are responsible for all statements made in their work and for obtaining permission to reprint or adapt a copyrighted table 
or figure, or to quote at length from a copyrighted work. Authors should contact original author(s) and original publisher to see if 
permission is required and to request nonexclusive world rights in all languages to use the material in the current article and in future 
editions. Include copies of all permissions and credit lines with the manuscript. (See p. 175 of the APA Manual for samples of credit 
lines.) 
Manuscript Submissions: Prepare manuscript in Microsoft Word format and submit an electronic copy to the Associate Editor, 
Meredith Ross (meredith.a.ross@vanderbilt.edu). 
Accepted Manuscripts: After manuscripts are accepted, authors are asked to sign and return copyright-transfer agreements. It is the 
responsibility of the contact author to ascertain that all co-authors approve the accepted manuscript and concur with its publication 
in the Journal. All accepted manuscripts, artwork, and photographs become property of the publisher. 
Production Notes: Files are copyedited and typeset into page proofs. Authors read proofs to correct errors and answer editor's queries. 
Authors may order reprints at that time. 
Peabody Journal of Education Online: More information about the PJE can be found through Vanderbilt University’s Peabody 
College web site at the following web address: http://peabody.vanderbilt.edu/Microsites/PJE/Peabody_Journal_of_Education.xml 
Illustrations: Illustrations submitted (line drawings, halftones, photos, photomicrographs, etc.) should be clean originals or digital 
files. Digital files are recommended for highest quality reproduction and should follow these guidelines: 

— 300 dpi or higher 

— Sized to fit on journal page 

— EPS, TIFF, or PSD format only 

— Submitted as separate files, not embedded in text files 
Color Reproduction. Color illustrations will be considered for publication; however, the author will be required to bear the full cost 
involved in color art reproduction. Color art can be purchased for online only conversion and reproduction or for print + online 
reproduction. Color reprints can only be ordered if print + online reproduction costs are paid. Rates for color art reproduction are: 
Online Only Reproduction: $225 for the first page of color; $100 per page for each of the next three pages of color. A maximum 
charge of $525 applies. Print + Online Reproduction: $900 for the first page of color; $450 per page for the next three pages of color. 
A custom quote will be provided for articles with more than 4 pages of color. 
Tables and Figures. Tables and figures (illustrations) should not be embedded in the text, but should be included as separate sheets 
or files. A short descriptive title should appear above each table with a clear legend and any footnotes suitably identi-fied below. All 
units must be included. Figures should be completely labeled, taking into account necessary size reduction. Captions should be typed, 
double-spaced, on a separate sheet. All original figures should be clearly marked in pencil on the reverse side with the number, author's 
name, and top edge indicated. 
Proofs and Reprints. Page proofs are sent to the designated author using Taylor & Francis’ Central Article Tracking System (CATS). 
They must be carefully checked and returned within 48 hours of receipt. Reprints of individual articles are available for order at the 
time authors review page proofs. A discount on reprints is available to authors who order before print publication. 


Peabody Journal of Education 


Issues of Leadership, Policy, and Organizations 


Volume 84, Number 1, 2009 


Introduction to the Special Issue on the Education Doctorate 1 
James W. Guthrie and David D. Marsh 


The Case for a Modern Doctor of Education Degree (Ed.D.): Multipurpose Education 


Doctorates No Longer Appropriate 3 


James W. Guthrie ee 


The Changing Context of K-12 Education Administration: Consequences for 
Ed.D. Program Design and Delivery 9 


Ellen Goldring and Patrick Schuermann 


Building, Sustaining, and Expanding the Education Doctorate at Peabody College: 


An Administrative View 44 
Catherine Gavin Loss 


From Curricular Alignment to the Culminating Project: The Peabody College 

Ed.D. Capstone 48 
Claire Smrekar and Kristin McGraner 

Re-envisioning the Professional Doctorate for Educational Leadership and Higher 

Education Leadership: Vanderbilt University’s Peabody College Ed.D. Program 61 
Timothy C. Caboni and Eve Proper 


Rethinking School Leadership Programs: The USC Ed.D. Program in Perspective 69 
David D. Marsh and Myron H. Dembo 


A Professional Doctorate in Educational Leadership: Saint Louis University’s 
Ed.D. Program 86 


Susan Toft Everson 


Strategies for Elevating the Public and Professional Regard of the Ed.D. 100 
James W. Guthrie and David D. Marsh 


MATT A 


0161956X (2009) 


olume 84 Issue2 ISSN 0161-956X 2009 


Peabody Journal - 
Education 


(issues of Leadership, Policy, and Organizations 





Routledge 


Taylor &Francis Group — - 





PEABODY JOURNAL OF EDUCATION 


Issues of Leadership, Policy, and Organizations 


Camilla P. Benbow, Dean, Peabody College 
James W. Guthrie, Editor 
Michele Thompson, Coordinating Editor 
Meredith Ross, Associate Editor 


EDITORIAL ADVISORY BOARD 


Darlyne Bailey, University of Minnesota David W. Gordon, Elk Grove Unified School District, California 
R. Dale Ballou, Peabody College of Vanderbilt University Eric A. Hanushek, Hoover Institution, Stanford University 

Sir Clive Booth, Oxford Brookes University, England Stephen P. Heyneman, Peabody College of Vanderbilt University 

Mary M. Brabeck, New York University Phillip W. Jones, The University of. Sydney, Australia 

Mark Bray, The University of Hong Kong, PRC Kazuo Kuroda, Waseda University, Japan 

Louis A. Castenell, Jr., The University of Georgia James H. Lytle, Trenton Public Schools, New Jersey 

David Y. Chen, Chongquin Technology and Business Michael K. McLendon, Peabody College of Vanderbilt University 
University (CTBU) Ann L. Marcus, New York University 

Mike Dishman, McLocklin, Murphy & Dishman, L.L.P. Bill McKinney, Region IV Education Service Center, Texas 

Lora Cohen-Vogel, Florida State University Francois Orivel, University of Bourgogne, France 

Joseph P. Du Cette, Temple University Penelope L. Peterson, Northwestern University 

Victor Firsov, Russian Academy of Teachers Retraining, Russia Michael J. Podgursky, University of. Missouri-Columbia 

Juan Froemel, LLECE/UNESCO, Chile Robert Slavin, Johns Hopkins University 

Louis M. Gomez, Northwestern University Jandhylala B. G. Tilak, National Institute of Education Planning 
Gerardo M. Gonzalez, Indiana University and Administration, India 


Abstracted and/or Indexed in: PsycINFO/Psychological Abstracts; Linguistics and Language Behavior Abstracts; Contents Pages 
in Education; Sociological Abstracts; Education Index; Education Abstracts; Social Services Abstracts; EBSCOhost Products; 
Cabell’s Directories; Family Index Database. 


Peabody Journal of Education (ISSN: 0161-956X) is published quarterly in February, May, August, and November for a total of 
4 issues per year by Taylor and Francis Group, LLC, 325 Chestnut Street, Suite 800, Philadelphia, PA 19106. 


US Postmaster: Please send address changes to Peabody Journal of Education, Taylor & Francis Group, LLC, 325 Chestnut 
Street, Suite 800, Philadelphia, PA 19106. 


Annual Subscription, Volume 84, 2009 
Print ISSN - 0161-956X, Online ISSN - 1532-7930 


Institutional subscribers: US $492, UK £294, €392 

Personal subscribers: US $62, UK £37, €49 

Institutional and individual subscriptions include access to the online version of the journal. Institutional subscriptions include 
access for any number of concurrent users across a local area network. Individual subscriptions are single username/ password only. 
Production and Advertising Office: 325 Chestnut Street, Suite 800, Philadelphia, PA 19106. Tel: 215-625-8900, Fax: 215-625-8563. 
Production Editor: Michelle McCarthy 

Subscription offices 

USA/North America: Taylor and Francis Group, LLC, 325 Chestnut Street, Suite 800, Philadelphia, PA 19106. Tel: 215-625-8900, 
Fax: 215-625-2940. 

UK/Europe: Taylor and Francis Customer Services, Sheepen Place, Colchester, Essex CO3 3LP, UK. Tel: +44 (0)-20-7017-5544, 
Fax: +44 (0)-20-7017-5198. 

For a complete guide to Taylor and Francis Group, LLC’s journal and book publishing programs, visit our 

website: www.taylorandfrancis.com 


Copyright © 2009 Taylor and Francis Group, LLC. All rights reserved. No part of this publication may be reproduced, 

stored, transmitted, or disseminated in any form or by any means without prior written permission from Taylor and Francis 

Group, LLC. Taylor and Francis Group, LLC grants authorization for individuals to photocopy copyright material for private 
research use. The publisher assumes no responsibility for any statements of fact or opinion expressed in the published papers. The 
appearance of advertising in this journal does not constitute an endorsement or approval by the publisher, the editor, or the 

editorial board of the quality or value of the product advertised or of the claims made for it by its manufacturer. 


Permissions. For further information, please visit: http://www.tandf.co.uk/journals/permissions.asp 


May 2009 


PEABODY JOURNAL OF EDUCATION, 84: 107—109, 2009 
Copyright © Taylor & Francis Group, LLC 

ISSN: 0161-956X print / 1532-7930 online 

DOI: 10.1080/016195609028 10062 


Routledge 


Taylor & Francis Group 


3907 LLNCY 


Introduction to the Special Issue on Informing the Future 
of School Choice Policy 


Casey D. Cobb 


University of Connecticut 


Robert Bifulco 


Syracuse University 


Courtney A. Bell 


Educational Testing Services 


In November of 2007, the Center for Education Policy Analysis at the University of Connecticut 
convened an interdisciplinary symposium entitled Public School Choice in a Post-Desegregation 
World: What Have We Learned and Where are We Going? The symposium brought together 
leading scholars from the fields of law, economics, sociology, education, and public policy 
to assess the future of public school choice programs in light of recent court rulings, policy 
developments, and research findings. The presentations included normative discussions of the 
legal constraints facing designers of school choice programs and empirical papers assessing the 
effects of school choice programs on student segregation, academic achievement, instructional 
practices, and other outcomes of interest. The articles in this special issue are by-products of that 
symposium. 

In the decades since the Supreme Court decision in Brown v. Board of Education (1954) ended 
de jure racial segregation in U.S. schools, attitudes about the potential for parental choice of 
schools to play a positive role in integrating schools have changed quite significantly. Choice 
programs served as a means for many White families to avoid racially integrated schools during 
the 1950s and the 1960s and, hence, represented a significant obstacle to racial integration. More 
recent decades, in contrast, witnessed the development of new choice programs, primarily in the 
form of magnet schools or controlled choice plans, intended to promote racial integration. 

In a recent decision, Parents Involved in Community Schools v. Seattle School District (2007), 
the Supreme Court has limited the use of race-based considerations in school assignments. In 
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addition to requiring changes in some of the controlled choice plans used around the country, 
this ruling raises the question of what role parental choice of schools will play in the future 
with respect to racial or other forms of school segregation. On one hand are those who fear that 
unfettered choice systems will increase school segregation. On the other are those who believe 
that choice programs can continue to play a positive role in.integrating schools and, indeed, may 
be the only option available in light of the current law. 

The recent Supreme Court ruling, decided by the narrowest of 5—4 votes, is by no means a 
radical break in the development of school desegregation law. Since the late 1980s, and perhaps 
even longer, the federal courts have been on a sustained retreat from school integration mandates 
and now have begun to extend limits on what measures can be taken to promote integration. 
These legal developments and the ideas motivating them, raise questions about the importance of 
racial and other forms of integration as a public policy goal. Certainly, the expansion of school 
choice programs over the last decade and a half is not the result of efforts to decrease segregation 
but rather has been motivated by calls to decrease bureaucratic constraints, increase market-based 
forms of accountability, encourage innovation, and thereby improve school performance and 
student achievement. Thus, when thinking about the design of school choice programs, it is worth 
asking how much importance should be placed on the goal of decreasing segregation versus other 
goals such as fostering innovation and improving student achievement. 

These questions and others are touched on and informed by the articles in this issue. The 
first three articles, by Koedel, Betts, Rice and Zau; by Bifulco, Ladd, and Ross; and by Holme 
and Richards, address the effects of programs to expand parent choice on segregation by race, 
class, and language group. Sorting out these effects poses several difficult empirical challenges, 
and these articles have assembled unique data sets from San Diego, California; Durham, North 
Carolina; and Denver, Colorado, respectively, and as a result are able to shed considerable light 
on this issue. 

The next three articles by Salvatore Saporito, by Courtney Bell, and by Claire Smrekar dig 
deeper and examine the motivations and decision-making processes of parents as they choose 
schools for their children. Together these articles highlight and provide nuanced accounts of how 
race and class influence these decisions. 

Two additional articles take up the issue of other goals that influence and need to be considered 
in the design of school choice programs. Janelle Scott reviews evidence on the performance of 
charter schools vis-a-vis student achievement, and in so doing suggests that effects on school 
segregation and on student achievement may be closely connected. Luis Huerta examines whether 
we should expect school choice, and particularly charter school programs, to foster instructional 
innovation. 

The concluding article, by Casey Cobb and Gene Glass, considers the findings from each of 
the empirical studies presented in this issue and considers the normative implications of those 
findings for the design of school choice policy. Cobb and Glass take heed on the question of how 
important of a goal integration should be and, echoing the opinion of Justice Stevens in the recent 
Parents Involved in Community Schools v. Seattle School District (2007) case, argue that we need 
to keep our eyes on the prize and maintain our concern with justice and our efforts to promote 
school integration. 

As editors, we would like to offer our sincere appreciation to the several individuals who served 
as blind reviewers, to the authors of these informative articles, and to the editorial board and staff 
of the PJE. We also extend our gratitude to the underwriters of the Post-Desegregation World 
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conference, including the Connecticut Department of Education, Magnet Schools of America, 
the Neag School of Education, and to many others who made that symposium a reality. Without 
such support this special issue would not be possible. 

We believe these articles offer insightful contributions to the empirical knowledge-base on 
school choice. Our hope is that they advance the conversation on the future of school choice 
among policymakers, educators, and scholars so that all children can benefit from what education 
has to offer. 
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We evaluate the integrating and segregating effects of three distinct school choice programs in San 
Diego. We go beyond the traditional question of racial integration and examine the integration of 
students by test scores, parental education levels, and language status. In addition to measuring the 
net integrative effects of school choice, we also examine the underlying motives behind student 
participation in school choice programs and the limiting influence of supply-side constraints. Two 
of the programs that we consider are rooted in 1970s integration-based reforms that provide public 
transportation for program participants. The third program is a state-mandated, open-enrollment 
program that requires participants to find their own transportation to and from their choice schools. 
We find that the two programs with underlying integrative objectives do indeed integrate the district, 
but the open-enrollment program segregates the district along most dimensions. Provision of busing 
and geographic preferences appear to be important factors in promoting integration. 


School choice is a prominent policy issue among education reformers. Proponents of school 
choice argue that it can increase educational output by improving match quality between stu- 
dents and schools and exerting competitive pressure among schools. In addition, because school 
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assignments are generally based on residential location, proponents argue that expanding school 
choice can help to break the link between segregation across neighborhoods and schools. Oppo- 
nents of school choice contend that it will result in a significant fraction of students, most likely 
the least advantaged, being left behind. Opponents argue that in leaving these students behind, 
school choice will further segregate schools along racial and socioeconomic lines. 

This article disentangles the effects of school choice on integration in the San Diego Unified 
School District (SDUSD), the second-largest school district in the state of California and the 
eighth largest nationwide. The school-choice program at SDUSD involves three components— 
the Voluntary Ethnic Enrollment Program (VEEP), the magnet program, and the open-enrollment 
program.' The former two programs are rooted in integration-based reforms dating back to the 
1970s, whereas the latter is a statewide program mandated in the 1990s.* Across the district, 
roughly one in five students participates in one of these three programs. 

The VEEP program was originally designed with the explicit goal of mixing White and non- 
White students to make within-school student populations more representative of the district’s 
overall diversity. Two structural aspects of the VEEP program suggest that it will apply integrating 
pressure to the district. First, the VEEP program uses busing patterns designed to move students 
from schools in predominantly non-White areas of the district to schools in predominantly 
White areas, and vice versa.* Students outside of the busing patterns cannot participate in VEEP. 
Second, and on a related note, the VEEP program provides public transportation for participants 
to and from their choice schools, which should increase the accessibility of the program for 
disadvantaged students. Prior to 1996, race was used as an explicit criterion for admittance into 
the VEEP program. However, after California’s passage of Proposition 209 in 1996, eligibility for 
the program is no longer linked to the race of the student. Although any student of any race can 
apply to attend any school in the VEEP busing pattern, the pattern is designed such that student- 
movement through the VEEP program should have a strong integrating effect on the district.4 

The magnet program was also designed with integration as an explicit objective. At its in- 
ception, the goal of the magnet program was to attract students from primarily White areas to 
primarily non-White areas by offering specialized curricula and additional resources such as 
reduced teacher—student ratios, teaching labs, field trips, and so on. Some magnets were also 
located to increase flows of non-White students to primarily White areas. Currently, the magnet 
program uses geographic clusters, determined by socioeconomic composition, to determine 


‘At the time of our study, charter schools provided a small but growing form of school choice, with about 5.5% 
of students enrolling in these schools (compared to 6.4% in VEEP, 6.6% in open enrollment, and 6.2% of students in 
nonresident magnet programs). We do not include charter schools in the present analysis because data on applications are 
not centrally available as they are for the other three programs. 

2We briefly describe each program next. For a more detailed description with additional background information, see 
Zau and Betts (2005). 

3The district creates “allied patterns” for VEEP sending schools. Students whose local school is a sending school 
can apply to be bused at district expense to any receiving school in that school’s allied pattern. If eight or more students 
apply for a given combination, the district will provide busing. Each allied pattern combines schools with quite varied 
demographic backgrounds. 

4Our description applies to applications to the school choice programs for fall 2001, which is the source of our data. 
Since that time, some minor changes have occurred, such as the use of both VEEP and magnet programs to provide 
school choice as required under the federal No Child Left Behind (NCLB) law. Our impression is that even by 2006 or 
2007 NCLB Choice had done little to alter VEEP or magnet school choice or to increase participation in school choice 
overall. See Betts and Danenberg (2004) for a review of NCLB and its early impact on SDUSD. 
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program admittance. Specifically, top priority for admission is given to students who reside 
in the clusters that least resemble the magnet school in terms of student composition. Similarly 
to the VEEP program, transportation is provided by the district for magnet students. Although 
any student can apply to any magnet school, the structure of the magnet program is such that it 
is expected to exert integrating pressure on the district. 

The open-enrollment program (known somewhat confusingly as the Choice Program) is avail- 
able to students statewide and places no restrictions on student movement, conditional on space 
availability at the receiving school. That is, any student can apply to any school in the district 
through this program (or even outside of the district, although across-district applications are 
rare). Although the open-enrollment program is the least restrictive in terms of student options, 
it does not provide transportation for student participants. The expected effects of the open- 
enrollment program on districtwide integration are unclear ex ante.® On one hand, participants 
in this program may be more likely to come from families whose residential location options are 
limited financially, limiting their local school options. Students from these families could use the 
open-enrollment program to attend more desirable schools. To the extent that this is the case, 
it should increase integration districtwide as disadvantaged students relocate to schools in more 
advantaged neighborhoods. However, participation in the open-enrollment program may be con- 
strained by transportation costs for some families because the program does not provide busing. If 
transportation costs deter disadvantaged families from participating and students from relatively 
advantaged families predominantly use the open-enrollment program to change schools, it could 
increase segregation within SDUSD. 

We examine the effects of these three school choice programs on integration by race, student 
achievement, and English-learner (EL) status. (We also briefly touch upon integration by parental 
education status but do not highlight it because of data issues.) The race measure is most often the 
primary target of school-choice policy, as is the case here, but the latter measures also inform the 
integration debate. Further, in the 2007 U.S. Supreme Court rejection of race-based preferences 
in school assignment decisions in Seattle and Louisville, Justice Anthony M. Kennedy suggested 
that districts might want to place greater emphasis on promoting integration on socioeconomic 
rather than racial lines. Although the Court (and the public) were sharply divided on these 
integration cases, both sides appear to agree that if the provision of equity in education is an 
objective of school choice policy, student mixing along nonrace dimensions provides important 
information about whether the school choice programs are successful. 

We begin our analysis by describing the demand for school choice in San Diego using appli- 
cation data from the VEEP, magnet, and open-enrollment programs. In many schools and grades, 
however, the number of applications exceeds the receiving school’s supply of spaces available. 
We therefore examine how supply-side constraints limit the integrating effects of the school 
choice programs. Similarly, not all lottery winners enroll in their choice schools, so we also study 
how this further reduction in actual school switching affects integration. 

In brief, the application data suggest that students are interested in using school choice 
to change their peer groups to be of higher socioeconomic standing. Therefore, participation 


‘The district defines four geographic clusters. Note that within any given cluster all applicants are treated equally 
regardless of race or ethnicity. Priority depends on clusterwide differences between the local and magnet schools. 

Expectations about the integrating effects of the VEEP and magnet programs are also not entirely clear. Although 
the structures of these programs suggest that they are likely to integrate the district by race, integration along nonrace 
dimensions, which we also consider, may or may not occur. 
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by minority and disadvantaged students exerts integrating pressure on the district. However, 
participation by advantaged students, who are also seeking to improve the socioeconomic status 
of their peers, exerts segregating pressure on the district. Overall, the VEEP and magnet programs, 
originating from court orders to desegregate the district, serve to integrate SDUSD by race. These 
programs also integrate the district by student achievement and parental-education levels, and the 
VEEP program integrates the district by English-learner status. However, the open-enrollment 
program generally applies segregating pressure on the student population at SDUSD. In some 
cases, the segregating effects of the open-enrollment program are nonnegligible in magnitude. For 
example, for integration by student achievement, the segregating effect of the open-enrollment 
program is so strong that it dominates the combined integrating effects of the VEEP and magnet 
programs. 


LITERATURE REVIEW 


School choice has the potential to promote integration by breaking the link between neighborhood 
segregation and school segregation. However, if students with higher socioeconomic status are 
more likely to take advantage of school choice programs because of differences in preferences, 
access to information and transportation, or other factors, or if families choose to use school- 
choice programs to move their children to schools with similar students, school choice could 
increase segregation (Archbald, 2004; Bifulco, Ladd, & Ross, 2009; Gill, Timpane, Ross, & 
Brewer, 2001). Several researchers have noted that the effects of school choice on integration 
may depend on the structure of the particular school choice program (Gill, 2005; Gill et al., 2001; 
Henig, 1999; B. Levin, 1999; Teske & Schneider, 2001). 

Numerous studies indicate that choice participants are more likely to be White and of higher 
socioeconomic status (see Elmore & Fuller, 1996; H. Levin, 1998; Wells, 1993). In their reviews of 
the literature, both Henig (1999) and B. Levin (1999) find stronger evidence of differential partici- 
pation by parental-education level than by race. Studies looking at the types of choices that families 
make tend to indicate that parents seek schools where the composition of the student body more 
closely resembles the student’s own background (Gill et al., 2001; Henig, 1996; B. Levin, 1999). 
These findings suggest that choice programs may lead to segregation by socioeconomic status. 

Various authors have pointed out weaknesses in the existing empirical literature, which tends 
to focus on differences in student composition between districts or between choice and nonchoice 
schools. Gill et al. (2001) explained that studies that rely on a comparison of the demographic 
characteristics of participants and nonparticipants do not adequately address the issue of in- 
tegration. These studies ignore the fact that participants may opt into schools that are more 
homogeneous than their default schools (see also Archbald, 2004). When using data measured 
at the district level, Gill (2005) pointed out that although school choice programs may make 
district-level demographics appear more balanced by drawing privileged students from private 
schools and other districts, these students may be drawn to more homogenous schools within the 
district. Finally, Teske and Schneider (2001) claimed that “the central issue that requires much 
more careful study is linking stratification to specific forms of choice” (p. 625), and they stated 
that better empirical data on the effects of choice are needed. 

Our data from the SDUSD allow us to make a useful contribution to the empirical litera- 
ture on the important question of how choice programs affect within-district integration. Using 
student-level data, we can identify both the choice and local schools of individual applicants 
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rather than relying on comparisons between choice and non-choice schools or districts. Also, by 
looking at the effects of three separate choice programs within the same district, we are able to 
hold district characteristics constant and isolate the differential effects of each choice program. 
Last, we are able to measure separately the effects of the demand for and supply of school choice. 


DATA AND THE PROGRAM SELECTION PROCESS 


The data for this project are based on applications to the three centrally administered school choice 
programs at SDUSD for fall 2001, submitted to the district during the 2000-2001 school year. 
Our dataset includes student-level information on race, gender, parental education, and English- 
learner status; as well as test-score achievement (from the Stanford 9 exam) and information on 
activity in the choice programs. We also have detailed microlevel data from each school in the 
district which we use to assemble information on the student-body compositions of schools. This 
facilitates comparisons between each student’s local school and choice options. It also allows us 
to evaluate the integrative effects of school choice relative to a counterfactual where the school 
choice programs did not result in any student movement based on applications for the 2001-2002 
school year. 

Many of the applications to the school choice programs are missing information that we require 
to make inference about the integrative effects of school choice and the underlying demand for 
school choice. We omit these applications from our analysis. Our largest data omissions are 
because of students’ grade levels—we omit applications from students for kindergarten and the 
sixth grade. For kindergartners, who apply prior to enrollment at SDUSD, we do not have student- 
level demographic information, making it impossible to infer integrative effects. Applicants for 
the sixth grade are omitted because heterogeneity in the schooling structure at SDUSD obscures 
the choice sets for these students and, more generally, confounds our ability to identify the peer- 
group-based demand for school choice among these students. The structural issue here is that 
some elementary schools at SDUSD end after the fifth grade whereas others end after the sixth 
grade, and therefore students entering the sixth grade can use school choice to alter their schooling 
structure (between elementary and middle school). Because we are interested in the integration- 
based motives behind the selection of choice schools, this additional dimension of demand that 
is unique to sixth-grade students would confound our analysis. In addition to omitting students 
applying for entry into kindergarten and the sixth grade, we also omit applications where key 
information is not available, including basic demographics, local schools, or grade levels, among 
other things. 

We ultimately use about half of the total number of applications for each program in our 
analysis. Of the omitted applications, roughly three in five are omitted because they are for 
kindergarten or the sixth grade, and two in five for other reasons. The patterns of missing 
applications across the three programs are very similar, suggesting that comparisons across 
programs using our data set will be reasonable. The primary implication of our missing data is 
that we will understate the net integrative effects of the choice programs for this single year. Later 
in the analysis, we attempt to correct for this by scaling up our estimated integrative effects based 
on districtwide enrollment in the school choice programs.” 


’Detailed information about the application data omitted from our analysis is available from the authors upon request. 


INTEGRATING AND SEGREGATING EFFECTS OF SCHOOL 115 


TABLE 1 
Demographic Information from San Diego Unified Schoo! District and the Choice Programs for the 
2000-2001 School Year 
——<$———— ee ee ee SE a Se ee 





Race 
White Black Asian Hispanic § Below Median English 
(%) (%) (%) (%) Test (%)* Learner (%) Population? 

District overall Qe 16.1 17.5 38.4 50 wong 143,030 
VEEP® 

Sending schools 6.7 19.9 15.4 57.8 66.6 47.1 40,573 

Receiving schools 51.8 8.8 14.0 24.2 30.3 14.0 21,736 

All elementary schools 25.0 16.5 15.6 42.1 M2 ST 78,087 
Magnet@ 

Cluster | 50.7 OF] 12.0 26.3 B27) 11.3 22 02N 

Cluster 2 44.2 10.2 14.1 30.3 40.2 14.3 24,363 

Cluster 3 28.0 12°5 SES Ze 40.4 19.0 17,085 

Cluster 4 8.0 Nel 17.4 53:2 66.2 41.5 62,634 
Program participants 
Applications 

VEEP Sit 24.4 9) 58.2 65.4 41.2 2912 

Magnet OES 32.8 113 35.5 54.1 LOS 4,608 

Open enrollment 33.9 20.0 14.1 30.9 44.0 17.6 4,388 
Lottery winners 

VEEP Sal 17.4 15.9 61.5 60.9 40.9 885 

Magnet 22.6 28.7 12.6 350 46.2 18.1 1,679 

Open enrollment 45.9 12.6 122 21D 32.4 13.9 1,091 
Enrollers 

VEEP 4.0 15.9 ae 62.3 61.9 39:5 496 

Magnet 20.3 26.0 19.1 332 41.4 15.8 728 

Open enrollment 47.4 172.5) 133) 24.5 30.1 12.6 682 


Note. VEEP = Voluntary Ethnic Enrollment Program. 

“Median test score is based on the Stanford-9 reading scaled score; >There are some schools for special education 
students and continuation students that are not participants in the VEEP or magnet programs. These students are included 
in the overall population. Therefore the sum of VEEP and magnet students will not equal the total population. Also, 
not all elementary schools are VEEP participating schools. Finally, as noted in the text, our population of participants 
excludes data from some applications. More detailed information about the excluded applications is available from the 
authors upon request; ©Statistics based on entire populations at designated schools. VEEP percentages include students 
only from elementary schools. Because VEEP patterns are based on home elementary schools, it is difficult to discern 
from one cohort whether middle or high school students originated in VEEP participating elementary schools, although 
the majority of elementary schools do participate in VEEP. For example, for the population of students at a given high 
school, we do not necessarily know the elementary school to which each student would be attached. We therefore cannot 
identify the subpopulation of these students who are actually eligible to participate in the VEEP program. Despite this 
limitation, we expect that the group-level means at the elementary level are useful to show how the structure of the VEEP 
program should integrate students; “Statistics based on entire populations at all schools in designated clusters. Magnet 
percentages include all students who live in a magnet cluster feeder pattern. 


The top panel of Table 1 provides basic demographic information for the district along with 
relevant program-specific summary statistics.* Specifically, the table shows that VEEP sending 


8The bottom panel of the table provides information about school-choice participants. We discuss this information 
briefly next. 
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and receiving schools, and the magnet clusters, differ markedly in socioeconomic makeup (in 
the magnet program, this is particularly true when comparing Cluster 4 to the other clusters). 
The structures of the VEEP and magnet programs are suggestive of their expected integrative 
effects—increased student mixing between VEEP sending and receiving schools, and between 
clusters in the magnet program, should increase integration at SDUSD. Recall that unlike the 
VEEP and magnet programs, the open-enrollment program is unstructured. 


SUPPLY AND DEMAND FOR SCHOOL CHOICE 


In each program, students’ applications to relocate to choice schools are accepted based on 
space availability. If a school receives more applications than it has spaces available in a given 
grade, administrators admit students based on the “priority group” to which the students have 
been allocated, and within priority groups they use a lottery if the school has space for only 
some of the students in that priority group. Applicants are sorted into priority groups based 
on whether the student has a sibling who already attends the option school, the time of year 
in which the application was made (before or after the deadline), and whether the student is 
transferring from another school within SDUSD or (very rarely) from outside.’ As just described, 
the magnet program also uses geographic clusters to determine program admittance. For some 
magnet schools, applications are processed in strict order by cluster and then, within-cluster, by 
priority group. For others, fixed percentages of admittees must be accepted from each cluster and 
again, the above described priority groups are assigned within-cluster. 

In this section we consider the implied effects of the school choice programs on applicants’ peer 
groups based on applications, lottery outcomes, and enrollment decisions. The application data 
provide the clearest picture of the demand for school choice. However, the application analysis 
ignores important supply constraints. We use information on lottery outcomes to evaluate how 
supply constraints reduced the effects of the school choice programs on integration. Similarly, 
not all lottery winners enrolled, so we also consider how this further reduction in actual school 
switching affected integration. For clarity of exposition, we focus on the exposure of school choice 
participants to the group that is the most socioeconomically advantaged in each case. The bottom 
panel of Table 1 details demographic information about the composition of the applicant pool for 
each choice program. Demographics for lottery winners and enrollers are also provided.!° 


Integration by Race 


We first examine differences in the racial composition between applicants’ local and choice 
schools, where local schools are determined by students’ residential locations. Figure 1 illustrates 


°Tn addition the magnet program gives preference to students who are applying for continuity (i.e., the student was 
accepted into the magnet program in a similarly themed school in a lower grade span in the previous year). 

'0The bottom panel of Table | raises many interesting questions about the choice programs. For example, some students 
are more likely than others to win school-choice lotteries. This is likely a reflection of heterogeneity in school-choice 
preferences across student types. A more detailed analysis of these data is beyond the scope of this project. 
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the effects of the choice programs on integration between Whites and non-Whites at SDUSD.!! 
It shows the average difference in the percentage of White students between the choice and 
local schools of applicants for each racial/ethnic group, based on applications, lottery outcomes, 
and actual enrollment decisions. (In cases for which a student applied to more than one school 
in a given program we took a simple average of the characteristics of these schools. School 
demographics are calculated based on 2000-2001 information.). For example, the first bar in 
each trio in the figure shows the average percentage-point change in the share of White students 
at applicants’ schools that would be observed if all school-choice applications resulted in student 
movement. 

The figure shows that applicants to all three of the choice programs apply to schools that have 
a higher percentage of White students than their local school. As previously discussed, this means 
that integration is caused by non-Whites applying to choice schools with more White students, 
but that this pattern is somewhat offset by Whites who are also opting to apply to schools with 
more White students. However, because non- Whites apply in greater numbers than Whites, and in 
higher proportions than their shares of the overall student population (see Table 1, bottom panel), 
on balance the three choice programs should increase integration between Whites and non- Whites 
districtwide. Notably, the percentage increase of White students from applicants’ local to choice 
schools is especially high for the VEEP program (for applicants of all races). Because of this, 
and because applicants to the VEEP program are mostly non-White, the application data suggest 
that the VEEP program will play a large role in integrating Whites and non-Whites in the district. 

Of particular relevance for the debate on how deeply school choice affects racial integration 
is the open-enrollment program, because in this program any student can apply to any school 
in the district, except for magnet schools. Of the three programs, the open-enrollment program 
is closest to an unregulated market. Across all racial/ethnic groups, there is a clear pattern 
showing that applicants use the open-enrollment program similarly to the VEEP and magnet 
programs and apply to schools with a higher percentage of White students than their local 
schools. However, unlike in the VEEP and magnet programs, applicants to the open-enrollment 
program are disproportionately White, suggesting a segregating effect. 

The application patterns suggest that there will be decreased mixing among non- White ethnic- 
ities. Although all applicants are choosing schools that would increase their exposure to White 
students, they are also simultaneously applying to schools that would decrease their exposure to 
other racial/ethnic groups. We emphasize that this does not necessarily mean that students of all 
races and ethnicities actively make choices based on the racial/ethnic mix of schools, as many 
other factors are correlated with race and ethnicity. 

The application data will overstate the amount of student movement generated by the school 
choice programs because in many cases the number of applications exceeds the number of spaces 
available. To adjust our calculations for lottery outcomes, we include all students who apply to 
a given choice program but set the change in the racial mix they would experience to zero in 
any cases where all of their applications to a given program lost in the lottery process, on the 
presumption that students who do not gain admission stay at their local school. 

In Figure 1, the first bar in each trio of bars shows the changes that would result if all applicants 
won their lotteries and enrolled. The second bar shows the changes in racial/ethnic mixing that 


'l Ror the precise calculations on which Figure 1 is based, see Betts, Rice, Zau, Tang, and Koedel (2006). Betts et al. 
also provided detailed information about mixing among non-White ethnicities. 
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would occur if only winning applicants changed schools. It shows that supply constraints in the 
choice programs limit student mobility and decrease the implied changes in the racial/ethnic mix 
experienced by the applicant group. For example, the upper left panel of Figure 1 shows that if 
all were accepted, Black applicants to the VEEP program would experience an average change 
in the percentage of White students at their schools of 41.6%. However, the VEEP program’s 
supply constraint reduces the implied change in the percentage of White students at the schools 
of these Black applicants by 30.3 percentage points, down to just 11.3%. Because many of the 
applications are rejected, the average outcome experienced by the original group of applicants 
differs sharply from what the application data alone would suggest.'? 

Given that slightly fewer than one third of students are already in various choice programs in 
San Diego, we find this quite remarkable: The expressed demand for school choice in this district 
continues to outstrip the supply of slots, and hence the integrative effects of school choice are 
limited. 

In some cases, focusing on lottery winners rather than on all applications results in a directional 
change in the implied integrating effect of a given program. For example, White applicants to the 
magnet program appear, weakly, to apply to schools that are “more White.” However, when we 
focus on lottery winners, constraints based on program availability actually favor White students 
applying to schools that are “less White.” Therefore, Figure 1 shows that White lottery winners 
tend to win placement into schools that marginally reduce their exposure to other White students, 
on average. In this specific case, supply-side constraints reduce segregation between Whites and 
non-Whites, consistent with the original integrative objective of the magnet program. 

Finally, not all lottery winners end up accepting their offers to switch schools. The third bar 
in each trio of bars in Figure 1 depicts actual changes in racial/ethnic integration that result from 
school switching relative to the changes implied by the application and lottery-winner data. In 
this case, we set the change in the racial mix experienced by applicants to zero in any cases 
where they did not enroll in a school for a given choice program, regardless of lottery success. 
Focusing on those who actually enroll should reduce the implied change in the racial/ethnic mix 
compared to the lottery winner analysis.!* Figure 1 shows that the integrating effects of the three 
programs are lower than those implied by both the application data and the lottery winner data 
when conditioning on students who actually enroll in the schools to which they applied. 


!2 Although supply constraints will clearly limit the programs’ effects, we make two notes. First, if all applications 
were accepted and resulted in student movement, the peer-group change experienced by student movers would be smaller 
than is implied by the application data because more non-White movers would be part of each applicant’s new peer 
group. Second, the reported counterfactual requires that the resident student populations at the choice schools would 
not respond to such an inflow of students. Nationally, research suggests that resident students might indeed react. For 
example, Fairlie and Resch (2002) provided some evidence that White students tend to “flee” to private schools in areas 
with large numbers of disadvantaged non-White students in public schools. In related work on residential segregation, 
Card, Mas, and Rothstein (2008) found evidence of tipping points in terms of the percentage of the neighborhood made 
up of minorities. Below a certain point, changes in the percent minority have no effect, but beyond a certain point, Whites 
appear to leave the neighborhood. 

\3Conditioning on enrollment could also magnify the changes in racial/ethnic makeup implied by the lottery-winner 
data. This would be the case if the students who choose not to enroll are those who win lotteries at schools where the 
racial/ethnic differences between their choice school and their local school are of the opposite direction of the group 
as a whole. We perform an auxiliary analysis that uncovers little evidence of such behavior. That is, when we drop 
all applications that are not accepted, and subsequently all applications that do not result in an enrollment, the implied 
changes in peer groups for accepted applications are very similar to those for enrolled students. 
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Integration by Test Score Performance 


Next, we evaluate the integrating effects of the school-choice programs by student achievement 
as measured by test-score performance on the Stanford 9 standardized test in 2000-2001 .'4 
Specifically, we compare the mean percentage difference in the number of students performing 
above the district median at the local and option schools in the given grade. Again, we extend our 
analysis to look at lottery winners and students who actually enroll. 

Figure 2 shows the average difference in the percentage of students whose test scores are above 
the median between applicants’ local and option schools based on applications, lottery outcomes, 
and actual enrollment. Again, in cases where a student applied to more than one school ina 
given program we took a simple average of the characteristics of the schools to which the student 
applied. Across all programs and looking at both above- and below-median performers, Figure 2 
shows that students use the choice programs to apply to schools where there is a greater proportion 
of above-median performers. The differences between the local and option schools are again the 
largest for students in the VEEP program relative to magnet and open-enrollment. When we focus 
on lottery winners and then on those who actually attend, the average magnitudes of the program 
effects decline, broadly mirroring the results presented for race and ethnicity. These changes in 
magnitudes show the integration-based implications of the gap between the demand for school 
choice and the supply of slots in receiving schools.!° 

We use the number of applications by student test group to determine whether low-scoring 
students are more or less likely to apply to a given program (see Table 1). For the VEEP and magnet 
programs, the demand for choice appears to be strongest among students who are themselves 
below-median performers, with 65 and 54% of the applicants to these programs, respectively, 
scoring below the median. With the open enrollment program, the converse appears to be the 
case: Only 44% of the applicants are below-median performers. That is, below-median performers 
are overrepresented as applicants to the VEEP and magnet programs and underrepresented as 
applicants to the open-enrollment program. 


Integration by Language Status 


Finally, we consider the role of the choice programs in integrating students who are and are not 
fluent in English. Figure 3 shows the mean percentage difference between the local and option 
schools of applicants in the number of students who are fluent in English, based on applications, 
lottery outcomes, and actual enrollment. In all cases, students apply to schools that have a lower 
proportion of English Learners. Figure 3 shows reductions in the magnitudes of the effects implied 
by the application data after adjusting first for lottery outcomes and then for students who actually 
enroll. When conditioning on lottery winners and then on students who enroll, the VEEP program 
generates the largest average difference between the choice and local schools of applicants, and 
the open-enrollment program produces the smallest differences. 


\4For this analysis, we focus only on applicants who have taken the Stanford 9 standardized test. 
1S Although again, with the caveats noted in Footnote 12, our simple comparison here is likely to overstate the 
general-equilibrium magnitudes of the supply-constraint effects. 
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Integration by Parental Education 


Because there is a sizeable group of students in SDUSD for whom we do not have parental 
education data, we do not show results for this disaggregation. However, when we divided 
students based on whether their parents have obtained a high school diploma or less versus those 
with postsecondary education, the results were highly consistent with the foregoing results. In all 
programs, applicants apply to schools where a higher percentage of the students have more highly 
educated parents. The VEEP program again shows the largest implied changes for applicants of 
all parental-education types. As in the other cases, the effects of the choice programs implied by 
the application data alone are dampened considerably once we enforce the supply constraints and 
examine actual enrollment data. 


NET EFFECTS OF SCHOOL CHOICE ON INTEGRATION 


Although the previous analysis is informative, it cannot by itself give information about whether 
school choice programs lead to greater or lesser integration, nor can it speak to the relative 
magnitudes of each program’s integrating or segregating effects. We therefore extend our analysis 
to examine the effects of the three choice programs on districtwide intergroup exposure in San 
Diego schools. To do this, we use exposure indices. As an example, consider the exposure of 
Black students to White students in the district. An exposure index reports, for the typical Black 
student in the district, for example, the proportion of students in his or her school who are White. 
The formula for such an index is as follows: 


(+) 


where x; is the number of Black students at school j, X is the total number of Black students in 
the district, y; is the number of White students at school j, and f; is the total population at school 
j. Thus, the exposure index is a weighted average of the proportion of students who are White at 
each school, with the schools’ shares of the overall Black population serving as the weights. 

We compare the actual districtwide exposure indices to counterfactual indices that we calculate 
by “undoing” all of the student movement associated with the school choice programs. The dif- 
ferences between the two measures indicate the single-year effects of the school-choice programs 
on integration.'© Of course, we do not expect applications for a single year to change markedly 
the exposure of one group of students to another. It is the sum of many years of applications and 
subsequent school switches that determines how the overall mix of students changes. However, 
we use this snapshot of 1 year’s worth of school moves because for this subset of school choice 





16A¢ discussed in the Data and the Program Selection Process section, our single-year effects will be understated 
because we are forced to omit a sizeable fraction of the application data. We make an adjustment for this next when we 
extend our results to evaluate the net integrative effects of school choice in San Diego. 
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participants we know exactly where they would have gone to school if they had not enrolled in 
any of the choice programs.'’ This allows for the accurate calculation of the counterfactual. 

Figure 4 shows districtwide changes in exposure for each of our three measures of integration 
as well as the net change in exposure resulting from the combination of all the programs. As in the 
previous section, Figure 4 shows our results for exposure to.the group that is expected to be the 
most socioeconomically privileged in each case. 18 As suggested by the previous section, the VEEP 
program has the strongest integrating effects across all measures of diversity, followed by the 
magnet program (which integrates along all dimensions except language status). In contrast, the 
open-enrollment program generally has a segregating effect on the district, most notably along 
the dimension of test-score achievement. 

The first panel of Figure 4 shows that the VEEP and magnet programs unambiguously increase 
the exposure of non-White students to White students, and vice versa. The open-enrollment 
program increases the exposure of White students to Asian students but segregates White students 
from Black and Hispanic students. Notably, Asian students at SDUSD are less likely to be 
disadvantaged than Hispanic or Black students and, in fact, Asian students have much higher 
test scores and grade point averages than either of these groups at SDUSD.!? Whereas the 
VEEP and magnet programs integrate disadvantaged minorities with advantaged groups, the 
open-enrollment program is counterproductive along this dimension. 

Also of interest are the integrating effects of the three programs among non-Whites. In an 
analysis we omit because of space constraints, for each race or ethnicity we calculate the initial 
and final levels of exposure to other groups and the changes induced by school choice for fall 2001. 
Although the choice programs integrate Whites and non-Whites, they have a clear segregating 
effect on the exposure of non-White students to each other. 

The second panel of Figure 4 shows district-wide changes in exposure based on student 
achievement. Although the VEEP and magnet programs apply integrating pressure along this 
dimension, the open-enrollment program strongly segregates above- and below-median perform- 
ers. Dominated by the negative effect of the open-enrollment program, the overall effect of all 
three programs on integration by student achievement is negative. 

In the third panel of Figure 4, our language-status exposure indices show that the net effect of 
the three choice programs is a clear decrease in the exposure of English Learners to non-English 
Learners, dominated by the effects of the magnet and open-enrollment programs. The VEEP 
program results in a small increase in EL to non-EL exposure when taken alone, but this effect is 
overcome by the negative changes in exposure created by the other two programs. 

Part of the explanation for the negative effect on exposure between EL and non EL-students 
lies in the program participation rates of EL students, reported in the bottom panel of Table 1. 
English Learners in SDUSD in 2000-2001 made up 27.4% of the student population. Based 
on the application data, the only program in which EL applications exceeded the EL share of 
the student population in 2000-2001 was the VEEP program, where 41.2% of applicants were 
English learners. In both the magnet and open-enrollment programs, EL students were largely 
underrepresented as applicants, making up just 19.5 and 17.6% of the applicant populations in 


17This assumes, of course, that students would not respond to the lack of availability of school choice by changing 
their residential-location decisions, or attending private schools. 

\8Ror detailed results from all of the exposure-index calculations, see Betts et al. (2006). 

!9Rurthermore, Asian participants in the choice programs also have higher grade point averages than Whites. 
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these two programs respectively. Because all applicants use choice programs to reduce their 
exposure to English Learners, the lack of participation by EL students themselves in these 
programs reduces their exposure to non-EL students overall. 

Finally, we can extend our estimates of the effects of the VEEP, magnet and open-enrollment 
programs based on our sample from this single-year cohort to evaluate the net effects of the 
programs on the student population at SDUSD. This analysis will provide reasonable estimates 
of the net districtwide effects of the choice programs under two assumptions. First, our observed 
cohort of applicants must be similar in demographics and school choice preferences to other 
applicant cohorts at SDUSD. Given the size of each cohort at SDUSD, where the district-wide 
population exceeds 140,000 students, this seems quite reasonable. Second, the distribution of 
available choice slots within each program faced by our observed cohort of applicants should 
be similar to the districtwide distribution of active slots. This condition would be violated if, for 
example, previous cohorts took up the most desirable school choice slots prior to program entry 
by our cohort. In this case, our observed cohort would only be active in the choice programs along 
the fringes of what was left over by previous cohorts, meaning that our observed cohort would be 
structurally restricted from responding to the choice programs in the same ways as prior cohorts. 

Although this scenario cannot be entirely ruled out, it seems unlikely for three reasons. First, 
because the schooling process is finite, even if some of the early entrants into the school choice 
programs took the most desirable slots, these students would not retain these slots indefinitely— 
they would graduate. Given a perpetual line of exiting school-choice participants, particularly 
desirable slots that were absorbed by previous cohorts should continually become available. Sec- 
ond, heterogeneity in preferences among students and parents over choice schools as documented 
by Betts et al. (2006) and Glazerman (1997) calls into question the very existence of a set of 
“clearly preferred” school-choice slots. Third, because the three choice programs have existed 
for several decades each, the system is roughly in equilibrium, meaning that there has not been a 
radical departure from typical enrollment patterns in recent years. 

If these three conditions are met, the net effects of the choice programs can be inferred by 
scaling up the estimates from Figure 4 by the districtwide numbers of students in each program. 
For example, let Jy denote the measured integrative effect of the VEEP program along some 
dimension based on student movement from our sample, Ny the number of observed movers in 
the VEEP program from our sample, and 7y the total number of students participating in the 
VEEP program at SDUSD in 2000-2001. An approximation of the total integrative effect of the 
VEEP program at SDUSD, /7y, can be calculated as Iry = ly * (H). We make an analogous 
calculation for each program along each measured dimension of integration and report our results 
in Table 2. 

Table 2 suggests that the choice programs have sizeable integrative (and segregative) effects on 
the district, For instance, actual Black-White exposure in San Diego was 0.19598, and together, 
the VEEP, magnet and open-enrollment programs increased the exposure of Blacks to Whites by 
0.024 + 0.012 — 0.002 = 0.034. Thus, these programs increased exposure by about 3.4 percentage 


20In principle we could investigate the distribution of available slots in our cohort and compare it to the distribution of 
slots among all current participants. However, because we do not have reliable data on students’ local schools outside of 
our cohort of study we cannot determine what options were available to these students, a particularly relevant problem in 
the VEEP and magnet programs. 
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TABLE 2 
Estimated Net Effects of School Choice on the Exposure of Advantaged Students to Disadvantaged 
Students at San Diego Unified School District 





District-Level Exposure VEEP Magnet Choice 





No. of observed movers in our data 496 728 682 


Program size (2000—2001)* 8,508 8,508 9,926 
Scaling factor TNS 11.69 14.55 
Exposure to White students 

Black students 0.19598 0.024 0.012 —0.002 

Asian students 0.24914 0.022 0.002 0.006 

Hispanic students 0.19432 0.036 0.005 —0.0001 
Exposure to above-median test-score performers 

Below-median test-score performers 0.41656 0.011 0.004 —0.028 
Exposure to non-English learners 

English learners 0.54736 0.002  —0.009 —0.010 





Note. Based on calculations from the 2001-2002 school-choice applicant pool. Results are reported as the 
raw changes in the exposure indices. District-level exposure indices are reported for comparison. VEEP = 
Voluntary Ethnic Enrollment Program. 

Based on population shares of roughly 6, 6, and 7% of the total student population in the VEEP, magnet, 
and open-enrollment programs, respectively, as reported by the district. 


points.”! Similarly, the programs increased the exposure of Hispanics to Whites by 4.1 percentage 
points. On the other hand, the programs decreased the exposure of below-median performers to 
above-median performers by 1.3 percentage points and of English learners to non-English learners 
by 1.7 percentage points. 


CONCLUSION 


Students (and their families) appear to use school choice programs to change their peer groups to 
be of higher socioeconomic standing. With regard to race, this means that applicants use the choice 
programs to attend schools that are “more White.” This does not mean that families necessarily 
make decisions based on race, because race is correlated with many other factors. For instance, 
applicants also appear to use these programs to attend schools that have a higher proportion of 
above-median test score performers, more students who have highly educated parents, and fewer 
students who are English Learners. Because all students use school choice programs to attend 
schools with more socioeconomically advantaged peers, participation by disadvantaged students 
tends to increase integration, whereas participation by advantaged students applies segregating 
pressure. 

Across all the choice programs and grade spans at SDUSD, the demand for school choice 
exceeds supply. Therefore, the extent to which students are able to use these programs to alter 
their school-level peer groups is limited by accessibility. By focusing on the applicant group for 


21 This is a big increase as in proportional terms it suggests that the Black-White exposure index is about 21% higher 
than it would have been without these programs. 
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each level of our analysis (race, student achievement and EL status), we show that supply-side 
constraints limit changes in the school-level peers of applicants. 

In addition to the supply shortages created by the lottery process itself, lottery winners who 
choose not to attend also limit the changes in the socioeconomic environments at option schools. 
Although it is unclear exactly why all lottery winners do not choose to attend their option schools, 
some simply leave the district. Of interest, among the lottery winners who do not attend their 
choice school the following year, just 4.2% of VEEP winners and 7.3% of magnet winners fail to 
do so because they have left the district. In the open-enrollment program, 21.0% of winners who 
did not accept the offer left the district. For the remaining groups, their motives for not attending 
their choice schools are unknown. 

In addition to presenting evidence on the net integrative effects of the three choice programs 
in San Diego, we also consider their separate effects. The VEEP program has the strongest 
integrating effects on the district along each measure of integration that we consider. The magnet 
program also integrates the district; however, its effects are less pronounced. The open-enrollment 
program segregates the district’s schools along most dimensions. 

There are two notable differences between the VEEP and magnet programs, which integrate 
the district, and the open-enrollment program. First, the integrating programs both restrict student 
movement in patterns that are explicitly designed to promote racial integration. These patterns 
are also aligned such that integration along other dimensions occurs. Of interest, the designs of 
these programs continue to promote integration despite the fact that neither race nor any other 
socioeconomic measure is used as a direct criterion for student selection into either program. 
Comparing the relative effects of the VEEP and magnet programs, the VEEP program is more 
successful as an integrative tool. 

The second difference between the integrating programs and the open-enrollment program is 
that the integrating programs provide busing for student participants. This seems to have a large 
effect on the demand for school choice by disadvantaged students. For example, although 73% of 
the district was non-White in 2000-2001, just 66% of applicants to the open-enrollment program 
were non-White. In fact, using each measure of socioeconomic status that we consider, disadvan- 
taged students are underrepresented in the open-enrollment program. This underrepresentation 
is particularly peculiar because (a) these same disadvantaged students are overrepresented in the 
VEEP and magnet programs” and (b) disadvantaged students should be more likely to participate 
in school choice based on the evidence presented here that students use choice programs to im- 
prove the socioeconomic standing of their peers. That is, neighborhood segregation implies that 
structurally, advantaged students as a group should have less interest in school choice programs 
because they are more likely to reside in an area where the local school is socioeconomically 
advantaged. 

We conclude with three implications for research and policy-making in this area. First, ex- 
amining integration beyond the traditional focus of race can yield both supporting evidence, and 
in some cases, entirely new insights. For example, SDUSD’s choice programs in the year we 
evaluate lead, overall, to greater integration between Whites and other racial/ethnic groups but 


22For example, 95% of applicants to the VEEP program are non-White. However, in addition to the VEEP and magnet 
programs providing public transportation, this difference is also likely to be partly driven by their integrative designs. 
Nonetheless, it suggests that the underrepresentation of disadvantaged students in the open-enrollment program is not the 
result of a general lack of interest from these students. 
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to slightly greater segregation based on student test scores and language status. Second, even 
in a district such as San Diego, where one fourth of students participated in the three choice 
programs we study, it would be a mistake to use actual participation in choice programs as a 
measure of the demand for school choice: student applications far outstrip supply. Third, details 
in the design of school choice programs appear to be crucial in determining their overall effects 
on integration. The disparate findings for the open-enrollment program relative to the other two 
programs suggest that the provision of busing and incorporation of geographic preferences do 
much to level the playing field in favor of the relatively disadvantaged. 
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Using student-level data from Durham, North Carolina, we examine the potential impact of school 
choice programs on the peer environments of students who remain in their geographically assigned 
schools. We examine whether the likelihood of opting out of one’s geographically assigned school 
differs across groups and compare the actual peer composition in neighborhood schools to what the 
peer composition in those schools would be under a counterfactual scenario in which all students attend 
their geographically assigned schools. We find that many advantaged students have used school choice 
programs in Durham to opt out of assigned schools with concentrations of disadvantaged students 
and to attend schools with higher achieving students. Comparisons of actual peer compositions with 
the counterfactual scenario indicate only small differences in peer composition for nonchoosers on 
average. More substantial differences in peer environment emerge, however, for students in schools 
with concentrations of disadvantaged students and schools located near choice schools attractive to 
high achievers. The results suggest that expansions of parental choice may have significant adverse 
effects on the peer environments of a particularly vulnerable group of students. 


INTRODUCTION 


Scholars and policymakers have long worried that programs designed to increase parents’ 
choices of schools for their children might increase the isolation of disadvantaged students. 
Two separate hypotheses underlie this concern. The first is that high-achieving students with 
educationally involved parents are more likely to take advantage of expanded school choices than 
disadvantaged students with less active parents, with the result being higher concentrations of the 
most motivated and able students in some schools. The second related hypothesis is that these 
more advantaged students are especially likely to opt out of schools with large concentrations of 
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disadvantaged students and low levels of achievement. As a result, students who remain in such 
schools will experience especially detrimental changes in their peer environment. 

A relatively large literature examines the characteristics of students who take advantage 
of choice. These studies suggest that across a wide variety of choice programs and contexts, 
advantaged students opt out of their assigned public school at significantly higher rates than 
disadvantaged students. Far fewer studies compare students who opt out to the students in the 
specific schools they would have attended under more restrictive policy regimes. As a result it 
is difficult to quantify the effects that choice programs might have on the peer environments of 
schools with concentrations of disadvantaged students. 

In this study we use data from Durham, North Carolina, to examine how school choice policies 
affect the peer environments of students who remain in their assigned public schools. Durham is 
typical of many countywide Southern school districts and offers several different school choice 
programs. We use data on elementary and middle school students, including information on home 
addresses, to examine who uses the Durham choice programs to opt out of their assigned school 
and to compare the peer composition of specific schools to what the peer composition of those 
schools would be if all students attended their assigned public school. 

The article is organized as follows. Section 2 reviews the results of empirical studies related 
to our research question. Section 3 describes the school choice policies in Durham and the data 
we use. Section 4 presents our analysis and findings, and Section 5 concludes. 


LITERATURE REVIEW 


Several studies examine the characteristics of students who opt out of public schools for private 
schools (Epple, Figlio, & Romano, 2004; Fairlee, 2006; Figlio & Stone, 2001; Long & Toma, 
1988; Lankford, Lee, & Wyckoff, 1995). These studies universally find that student ability, family 
income, and parent education are each positively correlated with the decision to attend private 
school. Several of these studies also find that White students are much more likely to enroll in 
private schools in metropolitan areas with large concentrations of Black students and that students 
generally are less likely to choose private schools in areas where the average level of achievement 
in public schools is higher. These results suggest private schools tend to cream skim, particularly 
in areas with concentrations of disadvantaged students. 

Other studies examine the characteristics of students who use vouchers to attend private 
schools. Hsieh and Urquiola (2003) focused on the nationwide voucher program in Chile and 
showed that wealthier families with higher education are much more likely to use vouchers to 
attend private schools. The most prominent voucher programs in the United States differ from 
that in Chile, however, by limiting eligibility to lower income families, restricting the ability of 
participating schools to select students, and/or prohibiting schools from charging tuition in excess 
of the voucher amounts. Evaluations of the voucher program in Milwaukee, which includes all 
of these restrictive provisions, find that together they can eliminate correlation between voucher 
use and income (Chakrabarti, 2006; Witte, 2000). Nonetheless, these studies find that parents 
with higher levels of education and involvement in their child’s education are more likely to take 
advantage of vouchers. Studies of voucher programs in Cleveland, New York City, Dayton, and 
Washington, DC, also find that parents who have higher levels of education are more likely to use 


132 R. BIFULCO, H. F. LADD, AND S. L. ROSS 


vouchers (Campbell, West, & Peterson, 2005; Howell, 2004; Howell & Peterson, 2002; Metcalf, 
West, Legan, Paul, & Boone, 2003). 

For a number of reasons, the studies of private school students and voucher users provide only 
limited information on how school choice policies are likely to affect the peer environments of 
nonchoosers. First, programs to expand parental choice options are designed to change the set 
of students who are able to take advantage of alternative school options. Thus, information on 
which students choose private schools may not provide a good indication of who will opt out of 
their assigned school under programs to expand school choice. Second, determining the effects 
of school choice on the peer environments of nonchoosers requires information on the schools 
that private school students or voucher users would most likely have attended in the absence of 
other options. Because this information is difficult to obtain, none of the studies of private school 
choice or voucher programs quantify the effects of choice options on the peer composition of 
specific schools. Third, the leading voucher programs in the United States restrict the number of 
voucher recipients to no more than 15% of the public school population (Campbell et al., 2005). 
In many places, much larger proportions of students use public school choice programs to opt 
out of their assigned school. 

Public school choice programs include magnet schools, charter schools, intra- and interdistrict 
open choice, and controlled choice. In contrast to publicly funded voucher programs that have been 
adopted in only a small number of places, some combination of public school choice programs are 
provided in most urban school districts in the nation. Studies of such programs provide some, but 
still limited, evidence of the effects of public school choice on the peer composition of students 
whose parents do not take advantage of expanded choice. 

Initially introduced as a way to desegregate schools, magnet school programs may now be the 
most prevalent form of school choice in the United States. Studies in several different cities report 
that parents who choose to enroll their children in magnet schools have higher levels of education 
and in some cases higher levels of income than parents who send their children to their assigned 
school (Archbald, 1996; Citizen’s Commission on Civil Rights, 1997; Martinez, Godwin, & 
Kemerers, 1996). In a study similar to ours, Saporito (2003) analyzed the enrollment patterns of 
students across 21 Philadelphia high schools. He found that nonpoor students from high-poverty 
areas and who are assigned to schools with below-average test scores are the most likely group 
to opt for magnets. However, he also finds that the magnet school program had only a small 
effect on the level of income segregation across Philadelphia high schools. Using a nationwide 
sample, Archbald (2004) compared levels of income segregation in districts with magnet schools 
to similar districts without magnet schools and does not find any association between magnet 
school programs and income segregation. However, even in cases where a magnet school program 
has little effect on district-wide measures of economic segregation, it may have substantial effects 
on the peer environment of particular schools. Thus, neither the Saporito (2003) nor the Archbald 
(2004) study provides a complete picture. 

Charter schools have been subject to intense scrutiny in recent years, but little of this research 
has carefully examined the effect of charter school programs on the peer composition of nearby 
public schools. Moreover, the available evidence generates mixed results. Booker, Zimmer, and 
Buddin (2005) followed samples of charter school students in Texas and in California who were 
observed in a regular public school a year earlier. In Texas, charter school students come from 
regular public schools with below-average achievement and were scoring substantially lower than 
their peers in those schools. Considering the strong preference it has given to authorizing charter 
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schools for at-risk students, however, the Texas charter school program might be exceptional. 
Their findings for California are more mixed. Although charter school students have lower test 
scores overall than their peers in the schools they previously attended, White students who select 
into charter schools tend to be higher achieving than the peers in the schools they left behind. 
Evidence from North Carolina indicates that college educated parents are more likely to enroll 
their children in charter schools (Bifulco & Ladd, 2007). 

Two recent studies provide valuable information on intradistrict open choice programs. Cullen, 
Jacob, and Levitt (2005) examined enrollment patterns across Chicago high schools where more 
than half the students opt out of their assigned neighborhood school. They reported that 74% of 
students in the top quartile on eighth-grade achievement tests opt out of their assigned school, 
and more than two thirds of those who opt out attend schools with high achievement levels. In 
comparison, only 37% of students in the bottom quartile opt out of their assigned school and 
the majority of those opt for public schools that have below-average levels of achievement. They 
also find that controlling for eighth-grade achievement levels, students whose parents have some 
college and who are more involved in their education are more likely to opt out, as are students 
who report higher grades, fewer changes of residence, and higher expectations for graduation 
on a survey administered in eighth grade. These results suggest that choice has led to extensive 
sorting by educational motivation and advantage in Chicago, but the study does not quantify the 
effects on the peer environments of those who do not opt out of their assigned school. Hastings, 
Kane, and Staiger (2006) studied an open choice program in Charlotte and found that parents of 
high-achieving students are much more likely to choose schools with high achievement levels 
and to send their children to a school far from their home than parents of low and moderate 
achieving students. Simulations based on such a distribution of preferences suggest that open 
choice substantially increases stratification by ability across schools relative to a neighborhood 
assignment plan. 

In sum, the evidence suggests that many forms of school choice generate greater concentrations 
of advantaged or high-achieving students in some schools. Most markedly, studies in many 
different contexts find that parents with higher levels of education are more likely to opt out 
of their assigned or neighborhood public school.! However, because most studies are unable to 
identify the schools choice students would have attended under alternative student assignment 
policies, effects on the peer composition of nonchoosers are much less clear. 


DATA 


Our analysis uses data on elementary and middle school students in Durham, North Carolina, from 
the 2002-2003 school year. The Durham Public School district is a countywide school system 
that serves approximately 32,000 students in 46 schools. The district is typical of the more than 
50 countywide Southern districts that serve a core city within a metropolitan area, and the results 
of the analyses that follow are most readily generalizable to these districts. Compared to these 
other Southern districts, Durham has slightly fewer students, a somewhat higher percentage of 





lfm addition to the U.S. studies just cited, studies of school choice policies in Scotland, England, France, the 
Netherlands, and New Zealand also find that students from advantaged backgrounds are more able to take advantage of 
expanded choice opportunities (Ambler, 1994; Fiske & Ladd, 2000; Willms & Echols, 1992). 
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minority students, but a relatively low percentage of students eligible for free lunch.” Unlike 
many other urban districts in the South, the Durham Public Schools have never been subject 
to a federal desegregation court order. Nonetheless, for a number of years the district used race 
conscious student assignment policies to promote racial balance. Since 1999, however, all student 
assignment decisions have been race blind. ’ 

The district relies primarily on contiguous, geographic attendance zones to determine student 
school assignments. There are 21 elementary school and 6 middle school zones. As is the case 
in many urban districts across the country, however, layered on top of the district’s geographic 
assignment plan are several programs that allow students to opt out of their assigned school to 
attend another public school. These programs include a long-standing policy that allows transfers 
between zoned schools for any reason and requires approval of any requested transfer provided 
space is available at the requested school and the student has an acceptable record of attendance and 
behavior; a magnet school program, established in the early 1990s, that during 2002-03 included 
six elementary and two middle magnet schools that offer educational programs and enrichment 
opportunities designed around a specific theme; three elementary and two middle schools that 
operate on a year-round calendar, which divides the year into 9 week quarters with a 3-week break 
between each quarter; and charter schools that are authorized and governed independently of the 
Durham Public Schools, including nine charter schools located in Durham that served students 
in Grades 3 through 8 during 2002-03.4 Each of these programs is by application only, and if 
there are more applications than seats available, admissions are determined by lottery.” 

The data for our analyses are drawn from two administrative sources: the North Carolina 
Department of Public Instruction’s End of Grade (EOG) test files and Durham Public School’s 
student transportation files. The EOG files contain a record for every public school student 
in Grades 3 through 8, including charter school students and students without test scores. In 
addition to information on test scores, the EOG files indicate what school the student attended, 
the student’s race/ethnicity and the highest level of education obtained by the student’s parents. 
The transportation files include an address for every student for which the district is responsible 
for providing transportation. These files were linked together and made available to us by the 
North Carolina Education Research Data Center. In addition, we obtained school attendance zone 
boundary files from the County of Durham that allow us to place individual addresses into school 
attendance zones. 


2 total of 55 county districts in the South serve a core city. Data from the 2002-03 Common Core of Data indicate 
that the median size of these districts is 40,514 and together they serve more than 3.5 million students. Among these 
districts, Durham ranks at the 80th percentile in percentage Black, the 75th percentile in percentage Hispanic, the 24th 
percentile in percentage White, and the 38th percentile in percentage free-lunch eligible. 

3The three elementary year-round schools and one of the middle school year-round schools each have a regular 
attendance zone similar to those for other zoned schools. However, students in those zones who do not want a year-round 
calendar are guaranteed admission at a neighboring school. The remaining seats in the year-round schools are available 
by application. One of the two year-round middle schools does not have a regular attendance zone and all admissions are 
by application. 

4The North Carolina charter school program was established in 1996 and the nine charter schools in Durham were 
first opened between 1997-98 and 2001-02. 

5Transfers under the transfer policy are granted on a first come, first served basis, and for admission to four of the 
elementary magnet schools, priority is given to students who reside in a small walk zone surrounding the schools. In all 
four of these magnets most of the seats go to students outside these small walk-zones. 
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Two issues in assembling these data are worth noting. First, because the Durham Public School 
district does not provide transportation for charter school students, we do not have addresses for 
the majority of charter school students residing in Durham during the years they attended charter 
schools. We do have current year addresses for students who transferred into a charter school 
during the school year. In addition, because we have transportation files for multiple years between 
1997-98 and 2005-06, we have addresses for charter school students who are observed at some 
point in other Durham public schools. Most of the charter school students between Grades 6 and 
8 during the 2002-03 school year had aged out of their charter school and transferred to a regular 
Durham public high school by 2005-06, and many of those that did not are observed in a Durham 
elementary school in earlier grades not served by their charter school. Thus, we were able to 
obtain addresses for 70% of charter school students in Grades 6 through 8. Because younger 
students are less likely to have aged out of their charter school by 2005-06, we have address data 
for only 45% of charter school students in Grades 3 through 5. The possibility that some charter 
school students might have moved between the year we observe their addresses and the year they 
enrolled in a charter school introduces some measurement error into our analysis, but the error is 
likely to be small. 

Second, one of the year-round schools, representing about one third of the middle school 
students who have chosen the year-round option, is housed in the same building as a traditional, 
zoned middle school. The student test score files do not distinguish students who attend the year- 
round school housed in this building from the students in the traditional program. To distinguish 
these two groups, we assume that any student in that school who does not live in the attendance 
zone for the traditional, zoned school is in the year-round program and that all other students in 
the building are in the traditional program. This process results in an assignment of approximately 
the correct total number of students to each school. Undoubtedly, however, some students are 
misclassified. For purposes of computing a student’s average peer characteristics, we treat the 
two schools as one. 

Table 1 presents summary information on the students in our study. Students in alternative 
schools are excluded from this table and all subsequent analyses.° The first and third columns 
provide information on all students in Grades 3 to 5 and Grades 6 to 8 in the EOG files. 
Approximately 60% of both elementary and middle school students attend their assigned school. 
Forty percent of elementary school students opt out of their assigned school—12.4% choose a 
magnet school, 15.8% transfer to another zoned school, 6.2% opt for a year-round school, and 
5.8% choose a charter school. Among middle school students, 14.2% choose one of the two 
magnet schools, 10.4% transfer to another zoned school, 10.4% choose one of the two year-round 
schools, and 5.2% attend a charter school. The lower portions of the table show that approximately 
60% of students are Black, 25 to 28% are White, and 7 to 9% are Hispanic. More than 38% of 
elementary school students and 48% of middle school students have a parent with a college 
degree. 

The second and last columns describe the sample of students for whom we have address data. 
Overall, we have addresses for 96.6% of the students in Grades 3 through 8 in 2002-03, although 
this percentage is lower for charter schools, particularly at the elementary school level. All the 


© Alternative schools include a school at the Duke Medical Center for students experiencing long-term hospital stays, 
and a school for students with behavioral or other issues. Assignment of these students is based neither on geographic 
assignment zones nor on parental decisions. 
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TABLE 1 ‘ 
Summary Information on Durham Students, 2002-03 
Grade 3-5 Grades 6-8 
Students With Students With 


All Students Address Data _ All Students Address Data 
PS Ped0 Sul OI a SS a eles 


Enrollment 
Total no. 8,049 7,715 WAZ 7,204 
In assigned school 4,813 4,747 4,298 4,236 
In a magnet school 997 991 1,024 1,019 
Tranferred to regular school* 1,274 1,274 747 T47 
Transferred to year-round school 495 492 751 T42 
In a charter school 465 De SH 261 
In an alternative school 5 0 224 199 
Race/Ethnicity 
% Black 60.5 60.1 60.9 60.7 
% White 25.2 25.4 27.8 28.1 
% Hispanic 8.9 on 6.8 6.7 
% Multiracial a 2.8 Des ZS 
% Asian 2) 2.4 1.8 1.8 
% Native American 0.3 0.3 0.2 0.2 
Parents’ education 
% Less than high school 8.3 8.3 6.5 6.2 
% High school 53.4 335 45.6 45.8 
% Two-year college 12.1 res 14.5 14.3 
% Four-year college 26.2 26.3 33 55a), 
Achievement level 
Average achievement —0.20 —0.19 —0.28 —0.27 
% Inconsistent mastery 2355 23.0 3221 31.6 
% Consistent mastery 47.8 47.8 42.2 42.3 
% Superior mastery 28.6 Do el 26.1 


Students who attend a zoned school other than the one to which they are assigned. Only students 
with address data could be identified as transfers. 


analyses that follow are conducted using the set of students for whom we have address data and 
exclude students attending alternative schools.’ This sample includes 7,715 students in Grades 3 
to 5 and 7,204 students in Grades 6 to 8. 


ANALYSIS AND RESULTS 


The concern that choice programs will increase the isolation of disadvantaged students rests on 
the hypothesis that advantaged students are more likely than disadvantaged students to use choice 
programs to opt out of their assigned public school, particularly when advantaged students are 





7Charter school students are more likely than other Durham students to have college educated parents, and thus 
excluding some charter school students from the analysis may lead to understatements of the extent to which school 
choice increases the isolation of disadvantaged students in assigned schools. 
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assigned to schools with concentrations of disadvantaged students and low levels of achievement. 
We test this hypothesis by comparing the characteristics of students who opt out of their assigned 
school with students who do not. We present both raw comparisons and multivariate analyses 
that compare the likelihood that different types of students will opt out controlling for where the 
student lives and characteristics of their assigned school. 

Next, we try to quantify the potential effect of choice programs on the peer environments of 
students who remain in their assigned public school, that is, those left behind. Specifically, we 
compare the actual peer environments of students who remain in their assigned school to what 
the peer environments for those students would look like if all children attended their assigned 
school. As we explain, such a comparison does not tell us the effect of school choice policies 
on the peer environment of nonchoosers, as the term “effect” is typically understood, but does 
provide insight into the potential of expanded school choice programs to alter peer environments. 


Who Opts Out? 


Table 2 examines the percentages of various groups of students who opt out of their assigned 
school. The first column reports the percentages of all students in each group who opt out. The 
second column indicates the percentages that opt out among those who live in an attendance zone 
with low levels of achievement. These are zones with mean EOG test scores at least one fourth 
of a standard deviation below the statewide mean and represent one third of the attendance zones 
at both the elementary and middle school levels. The third column indicates the percentage that 
opt out to attend a high achievement school. High achievement schools are those with average 
EOG test scores above the statewide mean. High achievement elementary schools include two 
of the three year-round schools and seven of the 18 zoned schools that follow a traditional 
calendar. At the middle school level this group includes the two magnet schools, one of the 
year-round schools, and one of the zoned schools. No charter schools are included among the 
high achievement schools. 

The first column shows that the percentages of students in Grades 3 to 5 who opt out of their 
assigned school do not differ significantly by level of parent education, and the percentage of 
low-achieving students who opt out is higher than the percentage of high-achieving students. In 
Grades 6 to 8, in contrast, a higher percentage of students with college educated parents opt out 
of their assigned school, and high achievers are more likely to opt out than low achievers. 

The marked differences between who opts out in the elementary school grades and who opts 
out in the middle school grades may reflect the composition of attendance zones. Because the 
district is divided into 21 elementary school attendance zones, but only 6 middle school zones, 
elementary schools draw from much smaller, more homogeneous areas than middle schools. As a 
result, the composition of student populations varies more widely across elementary school zones 
than across middle school zones. Among the elementary school zones, three have less than 20% 
of students with college educated parents, whereas three have higher than 60%. At the middle 
school level, the percentage of students with college educated parents only ranges from 42 to 
54%. Consequently, advantaged parents can more readily access schools with concentrations of 
advantaged parents and high levels of achievement through residential location, and thus have 
less reason to use choice programs to opt out of their assigned school at the elementary school 
level than at the middle school level. The residential location decisions of disadvantaged families, 
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TABLE 2 , 
Parcentage of Students in Who Opt Out of Their Assigned School, by Student 
Characteristics, 2002-03 


seis Min Wil 2A0c i ee aL 


From Low To High 
Total Achievement Zone* Achievement School? 
i 
Grades 3-5 
Parents’ education 
Less than high school 35.4 39.7* 2.4* 
High school 38.7 47.8 Ql 
Two-year college 42.1 54.3* 14.7* 
Four-year college Biee Oem DAs 
Student achievement 
Inconsistent Mastery 43.9* 49.2 Gye 
Consistent Mastery 38.4 46.6 ey 
Superior Performance 33.0* 53.4* A7eS* 
Grades 6-8 
Parents’ education 
Less than high school 35.1 37.5 14.0* 
High school 35.3 36.6 20.8 
Two-year college 43.9* 47.8* 263° 
Four-year college 44.0* 310m 34.2* 
Student achievement 
Inconsistent mastery Sou1F 83 16.4* 
Consistent mastery 39.1 45.8 27.0 
Superior performance 44.0* 54.4* 36.6* 


we EEE eee 


Note. EOG = End of Grade. 

a] ow-achievement zones are attendance zones with mean EOG scores at least one fourth 
of a standard deviation below the statewide mean. >High achievement schools are schools with 
mean EOG scores above the statewide mean. 

*Significantly different from the high school percentage for parent education variables and 
from consistent mastery percentage for student achievement variables in two-tailed f test at 
.05 level. 


however, are typically more constrained, and thus, they are less able than advantaged families to 
access their most desired elementary school through choice of residence. 

The second column of Table 2 focuses on students residing in low-achievement attendance 
zones. Among students in this group, children of college educated parents are significantly more 
likely to opt out of their assigned school than children of high school graduates, and the latter 
are more likely to opt out than children of high school dropouts. Also, among students residing 
in low-achievement attendance zones, high achievers are significantly more likely to opt out 
than lower achievers. These patterns appear in both the elementary and middle school grades 
but appear more strongly among the older students. The third column of Table 2 indicates that 
both elementary and middle school students whose parents have a 4-year college degree are 
substantially more likely than students whose parents have a high school diploma, or less, to opt 
out of their assigned school for a high achievement school. Also, high achieving students are 
more than twice as likely as low-achieving students to opt out for high achievement schools. 
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Multivariate Analysis of Who Opts Out 


Other factors in addition to the level of disadvantage in the assigned school might account for 
differences in the likelihood of opting out between groups. In particular, studies have shown that 
distance from home is an important factor for parents in choosing a school (Bifulco & Ladd, 
2007; Cullen et al., 2005; Hastings et al., 2006). We expect that the greater the distance between 
a student’s home and her assigned school, the more likely the student will be to opt out. Also, 
students with parents who value high achievement will be more likely to opt out of their assigned 
school if they live closer to a high achievement alternative. 

To compare the propensities of different groups to opt out of their assigned school controlling 
for the composition of the attendance zone in which they reside and the location of their homes 
relative to different schooling options we estimated linear probability models using an indicator 
of whether a student opted out of their assigned school as the dependent variable. Specifically, 
we model the decision to opt out as a function of student characteristics, the composition of the 
attendance zone in which the student resides, the distance of the student’s residence to his or 
her assigned school, and the distance of the student’s residence to the nearest high achievement 
school. 

The percentage of students with college educated parents is highly correlated with the per- 
centage of low achievers in an attendance zone. With only 21 elementary and 6 middle school 
attendance zones, it is not possible to separate the effects of these different aspects of peer com- 
position on the decision to opt out. Instead, we use a composite measure of the extent to which 
the students in an attendance zone are from disadvantaged groups. This disadvantage index is a 
simple average of the percentage of students whose parents do not have a college degree and the 
percentage of students achieving inconsistent reading mastery. 

The results of these estimations are presented in the first and third columns of Table 3. At 
both the elementary and middle school levels, Black students are more likely than White stu- 
dents to opt out of their assigned schools, a finding consistent with some other studies (Brunner, 
Imazeki, & Ross, 2006; Campbell et al., 2005). One possible explanation is that constraints 
on their residential choices make Black families less happy with their assigned schools than 
White families who are more able to move to the attendance zone of the school they most 
prefer. Note, however, that Hispanic students are considerably less likely than Black students 
to opt out of their assigned schools despite the fact that their residential choices may also 
be constrained. Bifulco, Cobb, and Bell (2008) reported a similar finding that Hispanic stu- 
dents are less likely to participate in interdistrict choice programs in Connecticut. These find- 
ings suggest that the choice process might differ considerably for Hispanics than for other 
groups. 

Conditional on attendance zone composition and the location of the student’s home, students 
with college educated parents are significantly more likely to opt out of their assigned schools than 
other students. In both the elementary and middle school grades, students with college educated 
parents are about 10 percentage points more likely to opt out than students whose parents are 
not college educated, a finding that consistently appears in the literature. At the elementary 
school level, low- and high-performing students are equally likely to opt out of their assigned 
schools, a result that is at odds with other studies which indicate that higher achieving students 
are more likely to opt out. At the middle school level, however, we do find that high-achieving 
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TABLE 3 
The Relationship Between Student Characteristics, Attendance Zone Characteristics, and the Decision to 
Opt Out 
Grades 3-5 Grades 6-8 
1 sd 3 4 
Black Omitted Omitted Omitted Omitted 
White —0.065* —0.056 —0.069** —0.062** 
(0.034) (0.035) (0.027) (0.027) 
Hispanic —0.168** —0.165** —0.093** —0.089** 
(0.040) (0.038) (0.035) (0.035) 
Other —0.069* —0.062* —0.041 —0.039 
(0.034) (0.034) (0.050) (0.050) 
College-educated parent 0.101** —0.131 0.096** —0.352 
(0.040) (0.115) (0.031) (0.177) 
Inconsistent reading mastery 0.023 0.025 —0.059 —0.059 
(0.015) (0.015) (0.047) (0.046) 
Consistent reading mastery Omitted Omitted Omitted Omitted 
Superior reading performance —0.006 —0.004 0.090** 0.094** 
(0.020) (0.020) (0.026) (0.026) 
Zone Disadvantage index* eso 0.956** one 2 
(0.129) (0.133) (0.556) (0.610) 
Zone Disadvantage x College-Educated Parent 0.472* 1.078** 
(0.245) (0.437) 
Distance from assigned school 0.063** 0.063** —0.006 —0.005 
(0.009) (0.008) (0.013) (0.013) 
Distance to nearest high-achievement school —0.046** —0.056** —0.028** —0.028** 
(0.007) (0.009) (0.007) (0.009) 
Distance to High-Achievement School x 0.020 —0.002 
College-Educated Parent (0.013) (0.012) 
No. of observations 7,411 TALI 6,955 6,955 


OT 


Note. Coefficient estimates are from Ordinary Least Squares estimation of linear probability model. An indicator of 
choice to opt out is the dependent variable. Robust standard errors adjusted for clustering by assigned attendance zone in 
parentheses. 

4 Average of percentage of parents with no college and percentage students showing inconsistent mastery in reading 
in the attendance zone to which the student is assigned. 

*Statistically significant at .10 level. “Statistically significant at .05 level. 


students are considerably more likely to opt out of their assigned school than low-achieving 
students.® 

Among both elementary and middle school students, the level of disadvantage in the assigned 
attendance zone significantly influences the likelihood of opting out. At the elementary school 
level, a 10-point increase in the zone disadvantage index increases the likelihood the student will 
opt out by approximately 11 percentage points. At the middle school level, a 10-point increase 


8In the analyses presented here, average performance on statewide reading tests are used as the measure of school 
achievement. Alternative estimates that used more complicated criteria of low, medium, and high performance based on 
math and reading test scores generate similar results. 
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in the disadvantage index increases the likelihood a student will opt out by approximately 17 
percentage points. 

According to Table 3, distance between home and school also affects the decision to opt out, 
especially for elementary school students. An elementary school student who lives | mile farther 
away from her assigned school is 6.3 percentage points more likely to opt out, and living 1 
mile further away from a high-achievement school decreases the likelihood of opting out by 4.6 
percentage points. Distance from the assigned school does not influence the decision to opt out at 
the middle school level. Although living closer to a high-achievement school does significantly 
increase the chances a middle school student will opt out, the effect is only half as large as it is 
for elementary school students. 

These patterns related to distances between home and school have clear implications for the 
student compositions of schools. To the extent that distance between home and their child’s school 
is an important consideration for parents of younger students, schools of choice at the elementary 
level are likely to draw most of their students from nearby neighborhoods. As a result, the ability 
of choice schools to integrate students from distant neighborhoods will be limited and choice 
schools will affect the student composition primarily of schools serving nearby neighborhoods. 
Because the distance between home and school appears less important for parents of older 
children, schools of choice at the middle and high school level may have more potential to attract 
students from distant neighborhoods and, consequently, may have larger effects on stratification 
across schools. 

In the second and fourth columns of Table 3 we allow the influence of disadvantage in the 
attendance zone where the student lives and the distance from a high achievement school on 
the likelihood of opting out to differ for parents with different levels of education. At both the 
elementary and middle school levels, the influence of zone disadvantage on the decision to opt 
out is significantly stronger for students with college educated parents than other parents. The 
influence of distance from a high-achievement school does not vary by parent education level. 

The estimates in the second column of Table 3 imply that the opt-out decisions of college 
educated parents are much more sensitive to the level of disadvantage in the assigned school 
than are the choices of other parents. The estimates in Table 3 imply that students with a college 
educated parent are 4.4% less likely than others to opt out of the least disadvantaged elementary 
school in our sample but 18.0% more likely than other students to opt out of the elementary 
school in our sample that has the highest level of disadvantage. Students with college educated 
parents are only 1.8% more likely than other students to opt out of the least disadvantaged middle 
school but are 17.5% more likely than other students to opt out of the most disadvantaged middle 
school.” 

In sum, the patterns of responses to the school choice program in Durham are similar to those 
that have been documented in many other contexts—compared to disadvantaged students and 
low achievers, advantaged students and high achievers are more likely to use choice to opt out 
of schools with concentrations of disadvantaged students to attend high achievement schools. In 
the middle school grades, students of college educated parents and higher achievers are more 





°The elementary school in our data with the lowest disadvantage index has 70% of students with college educated 
parents and fewer than 7% achieving less than consistent reading mastery. The elementary school with the highest level 
of disadvantage has 10% of students with college educated parents and 42% of students showing less than consistent 
reading mastery. The ranges of disadvantage are much more narrow at the middle school level. 
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likely to opt out than other students, and these groups of students are much more likely to opt out 
of schools with concentrations of disadvantaged students and low levels of achievement. In the 
elementary school grades, students with college educated parents and high achievers are no more 
likely than others to opt out of their assigned school. However, once we control for the level of 
disadvantage in the assigned school, students with college educated parents are more likely to 
opt out. Also, students with college educated parents are more likely than other parents to opt out 
of schools with high levels of disadvantage and for high achievement schools. 


A Counterfactual Comparison 


In this section we describe the overall effect of the pattern of choices just documented on 
the peer environments of various groups of students and schools. Specifically, we compare 
peer compositions given the actual distribution of Durham students across schools to the peer 
compositions that would have emerged if all students had attended the school to which they 
were assigned. This counterfactual, in which all students attend their assigned school, is probably 
not what we would observe in the absence of school choice programs. About one fourth of the 
students in Durham choose a school other than one of the schools with geographic assignment 
zones. If magnet, year-round, and charter schools were not available, more zoned schools would 
be required, and as a result attendance zones would be drawn differently. In addition, student 
assignment policies influence residential choices and decisions to opt out of the public schools in 
favor of private schools. Thus, we might observe a different pattern of residential segregation and 
private school enrollment, and more families exiting Durham County for more distant exurban 
areas, if Durham’s choice programs were removed. Nonetheless, the counterfactual comparisons 
presented in this section provide a good indication of the potential of choice to affect the peer 
composition of specific schools and groups of students. 

We begin by calculating a series of exposure indices given the observed distribution of students 
and comparing these to the same indices under the counterfactual. An exposure index measures 
the extent to which one group has contact with another, and can be defined for any pair of groups. 
Mathematically, the exposure rate is a weighted average of the percentage of Group B in each 
school, where shares of Group A are used as the weights. The exposure rates can be interpreted 
as the percentage of students in Group B in the school of the typical member of Group A. So for 
instance, the exposure of low-achieving students to college educated parents is the percentage 
of students with college educated parents in the typical low-achieving student’s school.!° Higher 
values indicate more exposure to the other group. These comparisons are presented in Tables 4 
and 5. The exposure rates presented in these tables are the rates for students who remain in their 
assigned school under the actual school choice policies in Durham. 

Table 4 shows exposure to students with college-educated parents. In the elementary school 
grades, actual exposure to students with college-educated parents is lower than under the counter- 
factual for all groups of students who remain in their geographically assigned school, except for 
students who themselves have college-educated parents. The decreases in exposure to college- 
educated parents relative to the counterfactual are larger for minority students than for White 


\0Rxposure of Group A to members of its own group can be interpreted as a measure of group isolation. In this case 
higher values represent greater isolation. 
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TABLE 4 
Exposure to College-Educated Parents, Actual Compared to Counterfactual 


Grades 3-5 Grades 6-8 


Actual Counterfactual Actual Counterfactual 





Student’s race ethnicity 


Black 34.9 37.6 42.9 47.6 
White 44.6 46.1 46.8 50.1 
Hispanic 34.3 37.6 44.9 48.6 
Parent’s education 
Less than high school 30.3 34.1 44.5 48.2 
High school 32.0 36.8 43.1 47.9 
Two-year college degree 43.0 42.6 44.1 48.1 
Four-year college degree 50.4 49.3 49.9 46.4 
Student’s achievement level 
Inconsistent mastery 31.9 35.6 42.8 47.3 
Consistent mastery 37.0 39.8 44.1 48.5 
Superior performance 44.1 45.3 46.8 50.2 


i 


Note. Actual averages are for students who remain in their assigned school and are 
compared to averages for these same students under the counterfactual that all students 
attend their assigned school. 


students, for students who parents are not college educated than for students who parents do have 
college degrees, and for low achievers than for high achievers. 

In the middle school grades, actual exposure to students with college-educated parents is 
lower than under the counterfactual for all groups who remain in their assigned school, including 
students with college-educated parents. Generally, the decreases in exposure relative to the 
counterfactual are larger at the middle school level than at the elementary school level. This result 
indicates that a relatively high percentage of students with college-educated parents opt out of their 
assigned middle school for magnet schools rather than for other zoned, middle schools. Across 
both elementary and middle school grades the largest decreases in exposure to college-educated 
parents is experienced by students whose parents have a high school diploma, but not a college 
degree—4.8 percentage points. Black and low-achieving middle school students also experience 
relatively large decreases in the percentage of classmates with college-educated parents. 

Table 5 compares the actual peer achievement levels of students who remain in their assigned 
schools to what those achievement levels would be if all students attended their assigned schools. 
If parent education is a good indicator of a student’s educational advantage, then, given the results 
in Table 4, we would expect to see that the actual peer achievement levels of students who remain in 
their assigned school are lower than under the counterfactual. Achievement levels might, of course, 
be influenced by the school one attends, and thus students’ actual achievement likely differs from 
the level of achievement they would have if all students attended their assigned school. Much 
evidence, however, suggests that even where differences across schools have important effects on 
learning rates, they account for only marginal amounts of the difference in achievement levels. 
Thus, examining peer achievement levels helps to confirm that less exposure to college-educated 
parents does in fact reflect less exposure to educationally advantaged students and to the learning 
opportunities that might be associated with attending school with higher achieving peers. 
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TABLE 5 
Average Achievement of Peers, Actual Compared to Counterfactual 


Average EOG Score Exposure to Exposure to Superior 
of Peers Low Performers Performers 











Actual Counterfactual Actual Counterfactual Actual Counterfactual 


ab Catt th ary ere te rte sects ol no~ cubs a eee ae ee oe 


Grades 3-5 
Student’s race ethnicity 
Black —0.371 —0.277 34.7 31.7 22.3 25.6 
White —0.248 —0.193 30.2 29.2 264 28 
Hispanic —0.376 —0.277 35.5 32.4 23.0 26.5 
Parent's education 
Less than high school —0.378 —0.290 35.8 32.7 22.9 25.7 
High school —0.356 —0.266 34.1 31.3 22.7 25.9 
Two-year college degree —0.340 —0.263 33.7 31.2 234 25.9 
Four-year college degree —0.283 —0.207 31.6 29.8 25.6 28.6 
Student’s achievement level 
Inconsistent mastery —0.403 —0.304 36.2 32.8 21.5 24.9 
Consistent mastery —0.322 —0.241 32.8 30.5 23.8 26.8 
Superior performance —0.249 —0.185 30.2 28.9 26.5 29.1 
Grades 6-8 
Student’s race ethnicity 
Black —0.244 —0.212 24.6 23.5 26.7 28.1 
White 0.022 0.017 17.2 17.3 36.8 37.0 
Hispanic —0.310 —0.259 27.2 25.5 24.1 26.5 
Parent’s education 
Less than high school —0.388 —0.321 30.1 27.6 21.6 243 
High school —0.221 —0.178 23.9 22.7 27.6 294 
Two-year college degree —0.151 —0.146 213 21.2 29.8 30.0 
Four-year college degree 0.013 —0.003 17.6 17.9 36.7 36.6 
Student’s achievement level 
Inconsistent mastery —0.309 —0.256 26.8 25.2 244 26.5 
Consistent mastery —0.189 —0.159 23.0 22.1 28.7 30.0 
Superior performance —0.015 —0.022 18.2 1s4 35.6 35.7 


ne ne 


Note. Actual averages are for students who remain in their assigned school and are compared to averages for these 
same students under the counterfactual that all students attend their assigned school. 


At the elementary school level, minority students, students whose parents are not college 
educated, and low achievers who remain in their assigned school all have classmates with lower 
average levels of achievement than they would under the counterfactual. These groups generally 
have average peer achievement levels approximately 0.09 standard deviations lower, percentages 
of low achievers just more than 3 points higher, and percentages of high achievers around 3 points 
lower than they would under the counterfactual. In contrast, actual peer achievement levels for 
White students, for students whose parents have a 4-year college degree, and for high achievers 
are greater, on average, than the peer achievement levels would be under the counterfactual. The 
differences between actual and counterfactual peer achievement levels are small for these groups. 

In the middle school grades, all groups of students who remain in their assigned school 
see lower levels of peer achievement than they would under the counterfactual, although the 


EFFECTS OF PUBLIC SCHOOL CHOICE 145 


0.2 : 
Y = -0.058 + 0.187 X Schools 


0.15 (0.041) (0.095) detailed in 


0.1 


0.05 





rile Sain lid 
-0.05 ° @ 


e 


Change in Disadvantage Index 





Disadvantage Index of Attendance Zone 


FIGURE 1 Change in Disadvantage Index. 


differences between the actual and the counterfactual peer environment tend to be greater for 
non-White students than for White students, for students whose parents are not college educated 
than for college-educated parents, and for low achievers than for high achievers. 

The comparisons in Tables 4 and 5 indicate that students who remain in their geographically 
assigned school, particularly those from disadvantaged groups, encounter schools with higher 
percentages of disadvantaged students and lower levels of achievement than they would if all 
students attended their assigned school.'! Even for disadvantaged student groups, the differences 
between the average peer compositions actually observed and those that would be observed under 
the counterfactual are small. It is important to note, however, that Tables 4 and 5 reflect averages 
for each group. These averages conceal considerable variation across schools. 

Figure | plots the distribution of changes in our index of disadvantage for each zoned school 
against the disadvantage index for the attendance zone assigned to that school. By change in the 
disadvantage index we mean the difference between the actual disadvantage index and that which 
would be observed if all students attended their assigned school. Here we group elementary and 
middle schools together. The plot reveals several things. 

Changes in peer composition vary across schools. Although the index of disadvantage increases 
for most schools, it decreases for at least eight schools. Schools that serve attendance zones with 
relatively low levels of disadvantage are much more likely than schools serving attendance zones 





1lWe have also compared measures of actual racial segregation with measures of racial segregation under the coun- 
terfactual. These comparisons are presented in another article that focuses more broadly on the potential effects of 
school choice in Durham on race and class-based segregation. These analyses indicate that actual measures of race and 
class-based segregation are greater than under the counterfactual that places all students in their assigned school but that 
differences in class-based segregation are more marked. See Bifulco, Ladd, and Ross (2009). 
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with higher levels of disadvantage to end up with lower percentages of disadvantaged students as 
a result of school choice policies. 

The three schools that serve attendance zones with the highest concentrations of disadvantaged 
students do not show especially large increases in their disadvantage index as a result of choice. 
Clearly, ceiling effects are at work here. There are so few. students from advantaged groups in 
these attendance zones that even if a large percentage of those students opt out, it results in only 
a small change in the overall peer composition of the schools. 

Even with these ceiling effects, a bivariate regression shows a statistically significant, positive 
relationship between the change in the disadvantage index and the disadvantage index of a 
school’s attendance zone. In other words, students in areas with substantial concentrations of 
disadvantaged students whose parents are unable or unwilling to take advantage of expanded 
school choice opportunities tend to see the largest declines in exposure to advantaged peers as a 
result of school choice policies. 

Two elementary schools (circled in Figure 1) show especially large changes in peer composition 
as a result of school choice policies. Details on the student composition of the attendance zones 
served by these two schools, as well as the changes in peer composition resulting from school 
choice, are detailed in Table 6. The changes in peer composition for these schools are substantial. 
The two schools show the percentage White drop from around 18% to less than 5% and the 
percentage college educated drop from 35.7 to 8.7% and from 25.5 to 12.5%. Both schools 
also see a substantial drop in peer achievement levels, including a reduction in the percentage 
of high-achieving students by nearly half. These examples show that even when the effects on 
average measures of peer composition are small, school choice policies can result in substantial 
changes in the peer environment of specific schools. 

Both these schools, as well as others that see relatively large differences between the actual 
and the counterfactual levels of disadvantage, fit a similar profile. They each serve an attendance 
zone with below-average percentages of White students, students with college-educated parents, 
and high-achieving students, but with enough students from these groups that their exodus in 
response to choice opportunities results in substantial changes in the school’s peer composition. 
Elementary School #1, in Table 6, also has a Montessori magnet school that is popular to many 


TABLE 6 
Schools with Largest Change in Student Composition 


a —__ 


Elementary School #1 — Elementary School #2 





% White in zone 18.1 17.8 
Change in % White —13.9 —14.9 
% college-educated parents in zone S57) 269) 
Change in % college-educated parents 27.0) —13.0 
Average EOG score in zone —0.132 —0.330 
Change in average EOG score —0.389 —0.291 
% inconsistent mastery in zone 24.3 27.6 
Change in % inconsistentent mastery 11.4 4.2 
% superior performance in zone 2901 Doe 


Change in % superior performance —14.3 —11.6 
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White, college-educated parents, located with the schools attendance zone. As a result, it loses 
an exceptionally large percentage of its high achieving students. ! 

The nonchoosers whose peer environments are changed the most by the choices of other 
students are not those who live in the most racially and economically isolated areas. These latter 
students would find themselves in schools with high concentrations of disadvantaged students 
whether or not parental choice policies are in place. Rather the students who see the largest 
changes as the result of school choice policies are those who live in attendance zones with 
substantial concentrations of disadvantaged students, but some share of advantaged students as 
well. The closer these zones are to choice schools that are attractive to high achieving students, 
the larger the effects of choice on peer achievement levels. 

It is quite possible that parents of advantaged students would, in the absence of the parental 
choice programs offered by Durham, use residential choice to leave these attendance zones. Thus, 
we cannot conclude that Durham’s choice programs have made the peer environment for students 
in these schools worse. Rather we interpret the results in Table 6 as upper bound estimates of the 
potential effect of parent choice programs on peer environments and believe they are sufficiently 
large to warrant further investigation. 


CONCLUSION 


In keeping with prior research, we find that advantaged students, specifically students whose 
parents have a college education, are more likely to opt out of their assigned schools. We 
also find that students with college-educated parents are most likely to opt out of schools with 
concentrations of disadvantaged students to attend schools with high levels of achievement. As 
a result, many neighborhood schools are left with fewer educationally advantaged students than 
under a counterfactual condition that places all students in their geographically assigned school. 
Schools with concentrations of disadvantaged students and schools located near choice schools 
attractive to high achievers show the largest difference between their actual student composition 
and that observed under the counterfactual condition. 

Several considerations limit the conclusions we can draw from these findings. First, one might 
question our ability to generalize findings from Durham. A persistent theme in the literature on 
school choice is that differences between who opts out and who remains behind depend heavily 
on context. Characteristics of neighborhood public schools and the local community, key features 
of school choice policies, the type of programs offered at alternative schools, the accessibility of 
information on school quality, and idiosyncratic historical and cultural factors can all influence 
the choices made by different groups of students (Figlio & Stone, 2001; Fuller, Elmore, & Orfield, 
1996). Our finding that distance between home and school is an important factor in parents’ choice 
of school for their children suggests that the effects of expanded parental choice are also likely to 
depend on the location of the available schooling options. Nonetheless, the student assignment 
policies in Durham, which overlay neighborhood assignment zones with several different school 
choice programs, are typical of the policies in many urban areas. Also, our findings concerning 
what type of students opt out of their assigned school, which schools lose the most students, and 





\2This effect is exacerbated by the fact that students who live in the immediate vicinity of the magnet school are given 
preference in the admission process. 
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which types of schools educationally advantaged students tend to choose, are consistent with 
findings from settings as diverse as Chicago, Philadelphia, Charlotte, Cincinnati, San Antonio, 
Milwaukee, Cleveland, Scotland, and New Zealand. 

More telling, questions can be raised about our counterfactual comparisons. School choice 
programs can influence how district officials draw attendance zone boundaries, parent choices 
about whether to keep their child in public schools, and patterns of residential location. Conse- 
quently, the peer environments students would encounter if all students attended their assigned 
school cannot be interpreted as the peer environments they would face in the absence of school 
choice policies. It is possible that if Durham’s school choice programs were eliminated, patterns 
of sorting similar to those documented here would reemerge as educationally advantaged families 
opt for private schools and move to areas associated with desirable schools. Thus, it is best to 
interpret the results of our analyses as estimates of the potential rather than the actual effects of 
public school choice policies on the peer environments of nonchoosers. 

Finally, it is not clear what effects different peer environments have on the academic achieve- 
ment and other important outcomes of individual students. Emerging evidence suggests that the 
most beneficial peer environment for a student depends on his or her background and ability 
level (Hoxby & Weingarth, 2005). Nonetheless, it is widely agreed that schools characterized by 
excessive concentrations of educationally disadvantaged students often have detrimental effects 
on student achievement. Any policies that result in more of those types of schools should cause 
concern among policymakers. 

Students who are assigned to poor performing schools and whose parents are unable or 
unwilling to take advantage of expanded school choice opportunities are an especially vulnerable 
population. The results of our analyses suggest that any benefits of expanded school choice 
that accrue to those able to take advantage of it might come at the expense of poorer learning 
environments for those left behind. 
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School Choice and Stratification in a Regional Context: 
Examining the Role of Inter-District Choice 


Jennifer Jellison Holme and Meredith P. Richards 
The University of Texas at Austin 


Despite the growing popularity of interdistrict choice plans over the past decade, the policy as- 
sumptions underlying their adoption have been subjected to very little empirical research. This study 
situates school choice within one metropolitan region, Denver, and examines the ways in which choice 
patterns relate to existing patterns of stratification between school districts. This regional focus offers 
insight into the patterns of interdistrict choice and the influence these patterns have on equity within 
a metropolitan region. Findings from this study indicate that relatively higher income students were 
more likely to take advantage of interdistrict choice, and that choice was more often used by students 
to exit from a less advantaged context (as measured by socioeconomic status) to a relatively more 
advantaged one. The article concludes with recommendations for policymakers in designing more 
equity-minded choice policies. 


School choice has been touted as “the civil rights issue of our time” (Peterson & Howell, 
2002, para. 1). Many believe, as Supreme Court Justice Clarence Thomas argued in the Zelman 
vs. Simmons-Harris decision upholding the Cleveland Voucher Program, that in this post-Brown 
era, school choice is the only means of “emancipation” for low-income students of color trapped 
in failing urban public schools (536 U.S. 639, 2002). No Child Left Behind (NCLB) choice was 
enacted based in large part upon this presumption: to ensure that “no child is trapped in a failing 
school” (U.S. Department of Education, 2001, para. 2), students attending schools that fail to 
meet performance targets for two consecutive years are granted the right to choose to transfer to 
a better performing public school. 

In practice, however, the freedom granted by NCLB school choice is limited, because it does 
not guarantee students the ability to choose schools outside their home districts (Holme & Wells, 
2008; Ryan & Heise, 2002). In the context of increasing segregation and concentrated poverty 
in the nation’s most troubled urban districts, choice plans that limit students’ choices to district 
boundaries leave many of the most disadvantaged students merely free to choose among other 
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segregated, high-poverty, and often struggling public schools (Holme & Wells, 2008; Ryan & 
Heise, 2002). 

These within-district restrictions have been blamed for the low participation rates under NCLB 
choice: Research has shown that nationally less than 1% of eligible students have taken advantage 
of the choice option under NCLB (RAND, 2008). Many assert that this low “demand” is because of 
the lack of available space in better performing schools within low-performing districts (Citizens’ 
Commission on Civil Rights, 2004). Although federal NCLB regulations encourage cooperative 
voluntary agreements between districts to expand choice options for students in districts with 
large numbers of failing schools, neither incentives nor funds are provided. As a result, very few 
of these agreements have been established (Holme & Wells, 2008). 

In an effort to rectify the within-district restrictions on choice, policymakers have proposed 
incorporating mandatory interdistrict open enrollment into the reauthorized NCLB legislation. 
Interdistrict open enrollment laws were enacted by a number of states in the 1990s as part of a 
growing movement to subject public schools to competitive market forces. Between 1993 and 
2000,' the number of states enacting interdistrict choice legislation expanded from 14 to 44, with 
enrollment in these plans expanding from 208,000 to more than 487,000 over the same period. 
By 2000, more students were enrolled in interdistrict choice programs than in charter schools and 
voucher programs combined (National Center for Education Statistics [NCES], 2002). 

Interdistrict choice, as with current NCLB choice policy, rests on two underlying policy 
assumptions: first, that exposing schools to the pressure of competitive market forces will bring 
about increased responsiveness, and therefore improvement, and second, that broadening choices 
will promote equity of opportunity by allowing students who lack the resources the freedom 
to transfer to a better performing public school. Despite the growing popularity of interdistrict 
choice plans over the past decade, the policy assumptions underlying their adoption have been 
subjected to very little empirical research. As a result, policymakers have very little information 
on which to rely in making judgments about the details any expanded NCLB school choice policy 
would require. 

This article examines the assumption that inter-district school choice promotes equity of 
educational opportunity through an analysis of inter-district open enrollment transfer patterns 
within one metropolitan area. Taken together, the data reveal a complicated story of a choice 
policy that, writ large, has contributed to existing regional inequality between districts. The 
results of this study offer important lessons for those who seek to include interdistrict choice in 
the reauthorization of NCLB. 


EXISTING RESEARCH: LIMITED DATA, LIMITED CONCLUSIONS 


Although the number of students exercising interdistrict school choice options has increased over 
the past several decades, researchers have been limited in their ability to examine interdistrict 
transfer patterns. Much of the problem lies in the inadequacy of state data systems, which have 
until recently lacked individual student identifier codes to track student choices. As a result, 
research to date has been restricted to analyses of aggregate (rather than student level) transfer 


'The year 2000 is the most current available national participation rate data on interdistrict choice (see NCES, 2002). 
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statistics among districts, data that lack detail necessary to elucidate sending and receiving patterns 
by student race, ethnicity, and socioeconomic status. 

These existing studies provide strong (although not conclusive) evidence that interdistrict 
choice serves to exacerbate between-district stratification by race and social class. The first form 
of such evidence comes from studies of participation rates,, which show that, in the aggregate, 
more affluent and White students disproportionately take advantage of interdistrict choice. State- 
level studies from New Jersey (Rutgers University, 2006), Massachusetts (Armor & Peiser, 
1997; Fossey, 1994), Georgia (Doering, 1998), and Wisconsin (Wisconsin Joint Legislative Audit 
Bureau, 2002) each concluded that, overall, White students and higher income students were 
more likely to participate in interdistrict choice than less affluent students and students of color. 
The second related source of evidence about the contribution of interdistrict choice to between- 
district stratification can be found in studies comparing the demographic characteristics of districts 
selected under open enrollment policies. These studies find that the districts that tend to “win” 
under interdistrict choice (i.e., are chosen by higher rates of transferring students) are those that 
serve relatively greater numbers of White and affluent students than districts that lose students 
under such policies (Armor & Peiser, 1997; Arsen, Plank & Sykes, 1999; Aud, 1999; Fossey, 
1994; MassInc, 2003; Rutgers, 2006; Wisconsin Joint Legislative Audit Bureau, 2002). 

This literature demonstrates that, in the aggregate, more affluent and White students participate 
in interdistrict choice and that more affluent and White districts tend to be chosen. However, few 
analyses to date have been able to link the two trends because the existing literature lacks concrete 
evidence as to which types of students are choosing which types of districts. 


THE MISSING LINK: STUDENT-LEVEL TRANSFER DATA IN A REGIONAL 
FRAMEWORK 


This article offers a perspective on the outcomes of interdistrict school choice policies that is 
distinctive in two ways. First, this study utilizes a unique data set that tracks the demographic 
characteristics of student transfers, which enables connections to be made between the charac- 
teristics of students and the districts they choose. Second, this study situates interdistrict choice 
within a local metropolitan context by connecting choice trends to existing patterns of racial, 
ethnic, and income stratification between school districts. Prior studies, by contrast, have largely 
treated sending and receiving districts as though they exist in isolation from others around them, 
either by focusing on the largest “net” sender or receiver districts within a particular state, or 
upon the largest sending/receiving pairs of districts (see Armor & Peiser, 1997; Fossey, 1994). 
This study utilizes a regional framework to understand how a competitive school choice policy 
such as interdistrict choice plays out in a context in which central cities and their surrounding 
communities and school systems are already in competition with one another for tax bases and 
middle-class housing demand (M. Orfield, 2002). Urban core and inner-ring suburban commu- 
nities are often the “losers” in this regional competition because they face declining shares of 
overall tax bases and house disproportionate shares of low-income residents (M. Orfield, 2002). 
Better-off and affluent communities start off ahead and easily accrue advantages with low tax 
rates, abundant resources, and sought-after school systems, which in turn generate high demand 
from middle-class families. As a result of this competition, the burdens of poverty and the re- 
sources to cope with those burdens are unequally distributed across communities and their school 
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TABLE 1 


Number of In, Out and Net Transfers by District Net Receiver/Sender Status 
rE a a a ee ala kar ci pede ie “at midi ciara | 





Transfers 
In Out Net Enrollment 
Net receivers 
Littleton 6 2,487 708 1,779 15,989 
Adams 12 Five Star Schools 4,034 2,644 1,390 37,341 
Jefferson County R-1 3,933 Lo 1,182 86,154 
Boulder Valley RE 2 1,893 738 1,155 28,171 
Englewood 1 927 584 343 3,495 
Sheridan 2 501 313 188 1,613 
Subtotal 13,775 7,738 6,037 172,763 
Net senders 
Douglas County RE 1 219 2,364 —2,145 50,370 
Denver Public Schools 4,491 6,319 —1,828 72,561 
Westminster 50 339 1,804 —1,465 10,683 
Adams-Arapahoe 28J 1,349 2,478 —1,129 33,831 
Adams County 14 473 1,394 —921 6,838 
St. Vrain Valley RE 1J B3i/ Asle72 —835 24,011 
Brighton 27J 363 695 —332 11,569 
Mapleton 1 539 808 —269 95 
Cherry Creek 5 1,749 1,765 —16 49,684 
Subtotal 9,859 18,799 —8,940 265,142 
Denver Metro Total 23,634 26,537 —2,903 437,905 





Note. Source: Colorado Department of Education. 


systems (Briggs, 2005; M. Orfield, 2002; Rusk, 2003). This study’s regional framework requires 
that interdistrict school choice be understood in light of, and situated within, these dynamics. 


THE SETTING: THE DENVER METROPOLITAN AREA 


The goal of this study is to examine interdistrict choice within one metropolitan region, Denver, 
and to examine how choice trends relate to existing patterns of stratification. As the most populous 
region in Colorado, serving more than 60% of all Colorado students, the Denver metro area is the 
“hub” of the state’s interdistrict choice activity (see Table 1). In the 2006-07 school year, more 
than 48% of the state’s interdistrict choice students transferred into one of the 15 school districts 
within the Denver metro area.” 

The Denver metro area provides an instructive context in which to examine interdistrict choice 
for several reasons. First, Colorado’s open enrollment law, which took effect in the 1994-95 
academic year, is representative of the laws in most other states in terms of its regulations and re- 
quirements. Like most states, Colorado’s law allows students to apply to attend any school district 
in the state tuition-free. Reflecting its origins in the movement to promote school improvement 


2In total, 48,543, or 6.1%, of the state’s 794,026 students elected to transfer into a Colorado district via interdistrict 
transfer. 
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through competition, Colorado’s interdistrict choice students are counted in the enrollment of 
receiving districts for the purposes of state aid.? Although Colorado’s law requires that all districts 
participate in open enrollment, school districts have a great deal of discretion as to the number 
and type of student transfers that they receive. Districts are free to deny transfers if the district 
determines that it lacks space, or if students fail to meet established eligibility requirements for 
particular programs including, “age requirements, course prerequisites, and required levels of per- 
formance” (Colorado Public Schools of Choice Act, 1990). As with most open enrollment laws, 
Colorado’s law allows districts to deny admissions that would be in violation of court-ordered 
desegregation plans, as long as the district is under court supervision. Although the Denver Pub- 
lic School District was under court supervision when the state’s law was passed, it was released 
from supervision in 1995 and the provision no longer applies (Lee, 2006). As with most other 
states, Colorado’s law also does not require receiving districts to provide transportation or give 
preference to low-income students or students from low-performing schools (Holme & Wells, 
2008).4 Given the similarity of the law to other states’ provisions, the operation of interdistrict 
choice in this context can be presumed to be representative of other states. 

Second, the Denver metropolitan area is roughly comparable to other large metropolitan areas 
on a variety of social and demographic characteristics. Although the metro houses relatively 
more Whites and Blacks and fewer Hispanics than other regions (U.S. Census Bureau, 2006),° 
the level of residential segregation between racial groups is comparable to other U.S. metros 
(Iceland, Weinberg, & Steinmetz, 2002). This study’s findings regarding the demographic impact 
of interdistrict choice, therefore, should not be attributable to the unique levels of segregation (or 
lack thereof) within the Denver metro area. 

Several other characteristics of the Denver metropolitan area are also worth noting. First, the 
Denver metropolitan area is, by virtue of its political geography, unlikely to have extraordinarily 
high levels of interdistrict choice activity compared with other metro areas. This is because of the 
relatively small number and larger geographic size of school districts in the region: The Denver 
metropolitan area contains just 15 school districts in six counties, encompassing 3,447 square 
miles (U.S. Census Bureau, 2000). By contrast, the far more “fragmented” Detroit metro region 
houses 83 separate school districts within its three counties (Macomb County Clerks Office, 
2007; Oakland Schools, 2007; Wayne RESA, 2008), encompassing 2,018 square miles (U.S. 
Census Bureau, 2000). As such, Denver has roughly one school district per 230 square miles, 
whereas Detroit has roughly one school district per 24 square miles, a nearly tenfold disparity 
that is only partly explained by the regional difference in population density. Because the Denver 
metropolitan area houses relatively fewer and geographically larger school districts, students in 
the area have relatively more intradistrict choice options. As a result, interdistrict choice activity 


3Under the Colorado Public School Finance Act of 1994, the total amount of money a district receives is based on 
a per-pupil formula comprising a base per-pupil allotment of $5,270.13 for each student enrolled on October | of the 
school year, plus adjustments for cost of living, personnel costs, district size, and number of students “at risk,” with a 
minimum guarantee of $6,550.19 per pupil (Colorado Public School Finance Unit, 2008). 

*Of the 43 states (including Puerto Rico) with interdistrict choice laws that were not designed specifically for 
desegregation purposes, only 2 require transportation assistance for all students and only 7 specifically require that 
low-income families be provided some form of transportation assistance (Holme & Wells, 2008). 

>Detailed tables comparing Denver city to the Denver—Aurora metropolitan statistical area and other metropolitan 


statistical areas on a variety of demographic indicators are available at the study Web site: http://www.edb.utexas.edu/ 
faculty/holme/ 
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would be predicted to be relatively lower than in regions that house more school districts per 
square mile. 

Second, like many other central cities, Denver is at a comparative disadvantage in the regional 
competition for tax dollars and middle-class families, with most new growth occurring outside 
the city boundaries. Although the city of Denver housed 29% of the metro area population in 
1990, its share of the population declined to 23.5% in 2006. Overall, the proportions of both 
Whites and Blacks in the city are lower than in the surrounding region (49.7% vs. 67.1% for 
Whites; 9.7% vs. 21.9% for Blacks), whereas the proportion of Hispanics is substantially higher 
than in outlying areas (34.8% vs. 5.2%). Relative to the greater metropolitan region, the city 
also has significantly higher poverty rates and child poverty rates (20.0% vs. 11.5% in poverty; 
29.4% vs. 16.1% children in poverty), higher proportions of adults with no high school diploma 
or GED (18.0% vs. 12.6%), and higher proportions of residents who do not speak English or do 
not speak English well (14.5% vs. 10.2%; U.S. Census Bureau, 2006). Denver, therefore, serves 
a significantly higher need population relative to its surrounding suburbs. Likewise, the racial 
and ethnic composition of the Denver Public Schools (DPS; coterminous with the boundaries of 
the city of Denver) reflects its struggle within the regional competition: the system serves a large 
number of students in poverty (67% in 2006-07), with a declining minority of Whites (20.4%), 
a growing majority of Hispanics (57.5%), and a moderate share of Blacks (17.8%; Colorado 
Department of Education, 2008). 

Yet the central-city DPS is not the only school district within the Denver metro area serving 
large concentrations of low-income youth. Although DPS has faced a shrinking White population, 
a dramatic increase in Hispanic enrollments and higher poverty student populations, other metro 
districts share DPS’s struggles. Two inner-ring suburban districts, Sheridan 2 and Adams County 
14, have similar levels of poverty and high concentrations of non-Whites. Yet just adjacent to 
DPS are also privileged districts, such as Douglas County RE | and Littleton 6, which have 
relatively low levels of poverty and low minority representation. Thus, as Myron Orfield (2002) 
observed, the picture of regional stratification is often quite complicated. So too, as this article 
demonstrates, is the picture of interdistrict school choice. 


DATA AND METHODS 


Data 


This analysis relies on interdistrict transfer data for the 2006-07 school year for the Denver 
metropolitan region obtained from the Colorado Department of Education. The Colorado data 
set has three specific advantages complementary to the purpose of this study. First, it allows the 
researcher to distinguish between open-enrollment transfers and other district transfers. Second, 
it links open-enrollment interdistrict transfer data to the student demographic characteristics of 
race/ethnicity and free- and reduced-price lunch status. Finally, it provides information regarding 


6 Although many other states maintain data on interdistrict transfers, it is often impossible to discern between genuine 
“open-enrollment” transfers and other types of student transfers, such as students who move, students who enroll in 
private schools or are rejoining the public system from private schools, or students who transfer to out-of-district charter 
schools. 
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which districts students transfer to, by sending district. As a result, this data set is one of the few 
that facilitates a detailed analysis of student flows among districts. 

Although the Colorado interdistrict transfer data have several distinct advantages, it also has 
several limitations. First, school-level data are not available, as student counts were too small to 
allow for privacy protections. Thus, care should be taken in interpreting these analyses, which 
are necessarily limited to describing district-level trends and do not provide insight into patterns 
of intradistrict activity. Second, because Colorado issues different official reports of interdistrict 
transfer data for aggregate transfer numbers and for transfers disaggregated by race/ethnicity 
and free and reduced-price lunch status, on several occasions Colorado’s official reports of the 
aggregate transfers slightly overestimated the summed disaggregated transfer data. Given the 
minor differences between these figures and the lack of any reason to suppose that the students 
with omitted data differ systematically from those with reported data, data were assumed to 
be missing at random. Addressing the question of what types of students chose which districts 
required the integration of these data sources. As such, each of these data sources were used 
and compared, with all calculations based on the appropriate denominators, to provide a more 
complete picture of the flow of students among districts. 

Third, because the state does not track student-level interdistrict transfer data, it is impossible 
to determine from the publicly available data whether students were counted in the demographics 
of their sending or receiving districts, or both. Given that transfer student demographics are 
hypothesized to vary systematically from sending and receiving district demographics, their 
inclusion or exclusion may slightly over- or underrepresent the magnitude of the difference 
between transfer student demographics and their sending and receiving districts, depending on 
the district in question. As noted previously, however, the relatively low proportion of each district 
population accounted for by student transfers, with transfers in and out accounting for 3.7% and 
7.1% of the Denver metro school population, respectively, should constrain any resulting biases 
in these analyses. 


Methods 


District-level data were first analyzed to determine which districts within the region were the 
largest senders of choice transfer students, which were the largest receivers of those students, 
and—because most districts send and receive transfers—which were the largest “net senders” 
and “net receivers” (Table 1). District data were then analyzed to determine how “net sender” 
and “net receiver” districts differed in terms of the student free and reduced-price lunch status 
(Table 2) and race/ethnicity (Table 3). To determine whether net senders and net receivers 
varied in terms of demographic characteristics of student transfers in, transfers out, and overall 
district demographics, transfer count data were analyzed via 2 x 2 contingency chi-squares 
using Pearson’s test for independence, with significant associations annotated in Tables | and 2. 
To provide an accessible measure of the practical significance of these effects, risk ratios were 
computed comparing the likelihood that transfers to and from net sending and net receiving 
districts qualified for free and reduced-price lunch status and fell into various racial and ethnic 
categories. 

Because the previous literature has often focused on net transfers, care was taken to compute 
and report these net transfer numbers. The regional framework adopted by this study, however, 
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TABLE 2 
Percentage and Number of In/Out Transfers Qualifying for Free/Reduced Price Lunch by 
Net Receiver/Sender Status 
———<—$——— a ea 
Qualify for Free/Reduced Price Lunch 


Transfers In Transfers Out 


% N % N District 
ee ee ee ee eee Pea 
Net receivers 


Littleton 6 8.2% 199 13.3% 83 14.5% 
Adams 12 Five Star Schools 11.5% 379 29.4% 758 28.8% 
Jefferson County R-1 25.0% 937 29.5% 767 24.1% 
Boulder Valley RE 2 14.1% 257 5.0% 32 17.8% 
Englewood 1 29.5% 251 33.3% 166 40.3% 
Sheridan 2 79.4% 395) 36.5% 92 72.9% 
Subtotal/Weighted Mean 18.9% 2,378 26.4% 1,898 24.0% 
Net senders 
Douglas County RE 1 0.0% 0 1.2% 27 4.3% 
Denver Public Schools 51.7% 2,134 43.4% 2,694 64.5% 
Westminster 50 51.9% Si 33.1% SD 69.3% 
Adams-Arapahoe 28J 17.7% 226 38.2% 902 53.9% 
Adams County 14 18.0% 66 22.4% 287 73.7% 
St. Vrain Valley RE 1J 0.0% 0 15.0% 160 28.6% 
Brighton 27J 18.6% 54 10.3% 59 28.4% 
Mapleton 1 47.5% 230 30.8% 223 49.8% 
Cherry Creek 5 16.9% 280 43.2% 732 19.7% 
Subtotal/Weighted Mean 35.3% 3,141 31.6% 5,659 38.6% 
Denver Metro Total 25.7% 5,519 30.1% 7,957 32.8% 
Senders vs. receivers 
Percentage-point difference 16.4% — 5.3% — 14.6% 
Risk ratio* List — OWES § — Me29%%5 





Note. Source: Colorado Department of Education. 

“For example, students transferring into net sending districts are 1.58 times as likely to qualify for 
free or reduced-price lunch as students transferring into net receiving districts (p < .01). Significance 
tests computed on values obtained via Pearson’s chi-square test of independence. 

Pr Deane 


mandates a focus on student flows among districts, which may be masked by an undue focus on net 
transfers. To discern district sending and receiving transfer patterns among districts, subsequent 
analyses focused on aggregate, rather than net, transfer numbers, via separate analyses of receivers 
and senders of interdistrict transfers within the region. Because of the complexity of the transfer 
patterns among sending and receiving districts, the top three senders and top three receivers 
of interdistrict transfers in the Colorado metropolitan region were selected for further analysis. 
Each of these senders and receivers were selected in terms of absolute (rather than net) transfer 
numbers. 

For each of the top three senders, the demographics of the transfers from each district were 
compared to those of the sending district and the receiving districts. Further, the demographics of 
the sending district were compared to those of the receiving districts. Likewise, for each of the top 
three receivers, the demographics of the transfers to each district were compared to those of the 
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TABLE 4 
Demographic Breakdown of Transfers from Denver Public Schools by Receiving District 


Transfers From Denver Public Schools 











Free/Reduced Lunch Rig Nor Whe SB 
Total Receiving Receiving 
Transfers Transfers District Transfers District 
Receiving District From DPS FromDPS vs.DPS  vs.DPS From DPS  vs.DPS_ vs. DPS 
Within Denver metropolitan area 
Jefferson County R-1 1,955 37.5% —26.9%** —40.4%** 50.5% —29.1%** —55.2%** 
Littleton 6 581 11.9% —52.6%** —49.9%*™* 27.6% —51.9%** —62.4%** 
Adams-Arapahoe 28J 428 43.0% —21.5%** —10.5%** 79.6% 0.1% —4.7%** 
Englewood 1 411 34.5% —29.9%** —24.2%** 51.8% —27.8%** —40.3%** 
Sheridan 2 372 75.3%  10.8%** 8.4%" 76.8%  —2.71% —5.8%** 
Cherry Creek 5 315 24.4% —40.0%** —44.7%** 41.0% —38.5%** —44.7%** 
Adams 12 Five Star Schools 175 17.1% —47.4%** —35.6%** 45.9% —33.7%** —41.2%** 
Adams County 14 117 42.1% —22.4%** 9.3%** 100.0% 20.4%** Seon 
Westminster 50 89 46.9% —17.6%** 4.8%** TAG ~ = 8.5% —5.5%** 
Mapleton 1 79 52.8% —11.7%* —14.7%*™* 100.0%  20.4%** —9.1%** 
Outside Denver metropolitan area 
Vilas RE-5 e555) 66.5% 2.0% —17.0%** 86.7% 7.2%** —17.2%"** 
Charter School Institute 147 18.8% —45.7%** —40.5%** 59.3% —20.3%** —42.0%** 
Other/Unknown 95 0.0%  —32.5%** — 0.0% —79.6%** — 
Total/Weighted mean 6,319 43.4% —20.9%  —27.1% 62.2% —-17.4%  —35.5% 
Risk ratio* — — 0.68** 0.49** — 0:78" Opie 


ee 2 Be i SS ee 


Note. Source: Colorado Department of Education. 

@For example, transfers from Denver Public Schools are on average 68 times as likely to qualify for free/reduced-price 
lunch as students in Denver Public Schools as a whole (p < .01). Significance tests computed on values obtained via 
Pearson’s chi-square test of independence. 

ope 05s ip Ue 


receiving district and the sending districts. In addition, the demographics of the receiving district 
were compared to the sending districts. Each comparison was evaluated, as above, via Pearson’s 
chi-square test for independence and associated risk ratios to determine the presence and strength 
of association between the demographic characteristics of transfers to and from each district 
in terms of free and reduced-price lunch status and race/ethnicity. Tables 4 and 5 summarize 
the findings for Denver Public Schools, which was both the top sender and top receiver of 
transfers.’ To illustrate the results of these analyses, regional maps were developed depicting 
the nw of students from DPS by free and reduced-price lunch and non-White status (Figures | 
and 2). 


TTables for Jefferson County and Adams 12 Five Star Schools are available at the study Web site. 
8Figures depicting the flow of students into DPS by free and reduced-price lunch and non-White status are available 
at the study Web site. 


SCHOOL CHOICE AND STRATIFICATION IN A REGIONAL CONTEXT 161 


TABLE 5 
Demographic Breakdown of Transfers to Denver Public Schools by Sending District 





Transfers to Denver Public Schools 





Free/Reduced Lunch Non-White 
Total VS. Sending VS. Sending 

Transfers Transfers Sending District Transfers Sending District 
Sending District to DPS to DPS District vs. DPS to DPS District vs. DPS 
Adams 12 Five Star Schools 192 50.0% 2M To — 39.690" 73.1% 34.8%** —41.2%** 
Adams County 14 85 56.8% —17.0%** 93%" 100.0% 16.7%** Bo 
Adams-Arapahoe 28J 1,020 46.6% —T4%*™* —10.5%** 81.1% 6.2%** —4.7%** 
Boulder Valley RE 2 20 0.0% —17.8%** —46.7%** 0.0% —22.8%** —56.7%** 
Brighton 27J 60 31.5% 3.0%** —36.0%*™* 67.3% 17.9%**  —30.1%** 
Cherry Creek 5 1,044 60.2% 40.4%** —44.7%** 66.5% 31.7%** —44.7%** 
Douglas County RE 1 da 0.0% —4.3%** —60.2%** 0.0%  —14.1%** —65.5%** 
Englewood 1 111 40.6% 0.3%"* —24.2%** 49.5% 10.2%** —40.3%** 
Jefferson County R-1 1,001 50.0% 26.0%** —40.4%** 67.0% 42.6%** —55.2%** 
Littleton 6 63 0.0% —14.5% —49.9%** 29.8% 12.7%** —62.4%** 
Mapleton 1 101 47.9% —1.9% —14.7%** 100.0% 295% 91%" 
Sheridan 2 74 67.2% —5.7% 8.4%** 100.0% 26.2% 5.8%" 
Westminster 50 357 68.9% —0.4% 4.8%** 90.1% 160% tee 55%" 
Other/Unknown 288 — — — 0.0% — — 
Total/Weighted mean 4,491 51.7% 15.0% —27.8% 70.9% 24.7% —32.1% 
Risk ratio* — — 1.96** 0.41** — 210575 0.44** 





Note. Source: Colorado Department of Education. 

For example, on average, transfers to Denver Public Schools are 1.96 times as likely to qualify for free/reduced-price 
lunch as the students in the district from which they are transferring (p < .01). Significance tests computed on values 
obtained via Pearson’s chi-square test of independence. 

p< Sep <.01.. 


FINDINGS: THE ROLE OF INTERDISTRICT CHOICE IN REGIONAL 
STRATIFICATION 


In presenting the results of our analysis, we focus first on the net transfer trends across the Denver 
region. In the second part of the findings section, we present more nuanced data on the region’s top 
three “sender” and “receiver” districts illustrating how transfer patterns relate to existing patterns 
of regional stratification. Although the data show that the net impact of interdistrict choice is 
relatively small with respect to the overall demographics of most individual school districts, our 
analyses suggest that interdistrict choice has exacerbated existing between-district stratification 


within the region. 


Choice in the Aggregate: Transfer Trends Across the Denver Metro Region 


As stated previously, the research on interdistrict choice to date has generally compared the charac- 
teristics of “net sender” and “net receiver” school districts in terms of demographic characteristics. 


162 J. J. HOLME AND M. P. RICHARDS 












VILASRES & 


BOULDER VALLEY RE 2 


Transfer Percent 
Free/Reduced Price Lunch 


0 — 20% 





20 — 40% 


40 — 60% CHERRY CREEK 5 


BEBE co— cox 
BE 20-00% 
4 —WESTMINSTER 50 
2—MAPLETON 1 


District Percent 
Free/Reduced Price Lunch 





ree 0— 20% 3 —SHERIDAN 2 

fi DOUGLAS COUNTY RE 1 4 —ENGLEWOOD 1 
20 — 40% 

BM 20 -c0% 

00-20% 

GE 20 - 100% 





FIGURE 1. Transfers from Denver Public Schools (DPS) by Free/Reduced Price Lunch Status. 


Taken together, this literature has shown that net receiving districts are, on balance, higher income 
and disproportionately more White than sending districts. As Tables 2 and 3 illustrate, trends 
for the Denver metro area corroborate this prior research: Net receiver districts on average had 
significantly smaller proportions of low-income, Black, and Hispanic students and significantly 
larger proportions of White students than net senders. 

The existing literature has also shown that, on average, higher income students are more likely 
to take advantage of interdistrict choice. Although our analyses corroborated this finding, the size 
of the effect was relatively small. To examine the overall composition of choosers, we discuss 
below the “transfers out” statistic to analyze the composition of students residing in the Denver 
metro opting out of their home district.2 As Table 2 illustrates, students opting out of their home 
district within the Denver metro area were only slightly less likely to be eligible for free- or 





°tm- and out-transfer numbers differ because many out-transfers transfer out of Denver metro school districts or 
transfer to virtual charter schools. 
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FIGURE 2 Transfers from Denver Public Schools (DPS) by Non-white Status. 


reduced-price lunch than the metro average (30.1% vs. 32.8%). This difference was statistically 
significant, however. 

Aggregate metro-wide transfer patterns by race/ethnicity revealed significant differences from 
prior research, which has found that, in the aggregate, White students are more likely to participate 
in interdistrict choice than students of color. In the Denver metro area, out-transfers were slightly 
less likely to be White than the metro district average and were more likely to be Black and 
Hispanic than students who did not transfer (see Table 3). 

A closer look at the data in Tables 1, 2, and 3 also illustrates how both net transfer numbers and 
aggregate transfer patterns mask a great deal of variation within these patterns of transfer activity, 
a phenomenon that has been largely overlooked in prior research on this issue. For example, 
some of the top “net senders” were predominantly White and low-poverty districts (e.g., Douglas 
County RE 1), while in other cases, some of the top “net receivers” were predominantly non- 
White and high-poverty districts (e.g., Sheridan 2). Further, many districts with lower net numbers 
had large absolute numbers of students transferring both in and out (e.g., Cherry Creek 5). This 
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variation illustrates the need to look beyond both net transfers and aggregate transfer numbers to 
closely examine the specific nature of transfer patterns in terms of absolute in and out transfers. 


Nuances in the Transfer Patterns: Senders, Receivers, and Regional Stratification 


In addition to the analysis of aggregate transfer trends, in-depth analyses were conducted on 
the top three “‘senders” and “receivers” to better understand the nuances of their transfer trends. 
Notably, the top three senders and receivers were the same districts: Denver Public Schools 
(housing the Denver central city), Jefferson County R-1 (immediately west of DPS), and Adams 
12 Five Star Schools (a suburban district north of, but not adjacent to, DPS). Together, these three 
districts accounted for 44.1% of all “out transfers” in the metro area, and received 55.0% of all 
‘sn transfers.” Next we focus on the “senders,” or the districts that lost students in interdistrict 
choice, examining the characteristics of the districts, of the students who left, and of the districts 
those students chose. A similar set of analyses follows for the “receivers,” or the districts that 
gained students in interdistrict choice. 


The Senders: The “Losers” in Interdistrict Choice 


The top three “senders” of students in interdistrict choice, ranked by absolute number of students 
transferring out, were (a) DPS, which lost 6,319 students; (b) Jefferson County R-1, which lost 
2,751 students; and (c) Adams 12 Five Star Schools (formerly Northglenn-Thornton District), 
which lost 2,644 students to interdistrict transfers. It should be noted that two of the three largest 
sender districts, Jefferson County and Adams 12 Five Star Schools, were in fact net receivers, 
receiving a net of 1,182 and 1,390 students, respectively. Both Jefferson County and Adams 12 
Five Star Schools are relatively low-poverty and low-minority districts. Denver Public Schools, 
by contrast, which is both high poverty and high minority, was a net sender, losing 6,319 students 
and suffering a net loss of 1,828 students. 


Sender #1: Denver Public Schools. Denver Public Schools, a largely non-White (75.3% 
Hispanic and Black, and just 20.4% White) and high-poverty (66.9% free and reduced-price 
lunch) district serving the Denver central city, was the top sender of interdistrict transfers. 
Although 71.6% of the DPS transfers attended other Denver metro area school districts, nearly 
one fourth of all transfers from Denver (1,555 students) attended Vilas RE-S, a virtual online 
charter school serving low-performing, at-risk youth. The plurality of all DPS transfers, 30.9%, 
attended schools in Jefferson County. 

As Table 4 illustrates, students transferring out of DPS were disproportionately more likely 
to be higher income and nonminority than DPS students overall. In addition, transfer students 
leaving DPS were roughly one third less likely to be eligible for free and reduced-price lunch 
than DPS students. Transfer students were also substantially less likely to be White and Hispanic 
than DPS students. Transfer students leaving DPS were also more likely to transfer to districts 
with student bodies that were higher income and lower minority. On average, transfer students 
from DPS were choosing districts with half as many students who were eligible for free and 
reduced-price lunch and half the minority population of DPS. 
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As shown in Figure 1, the variation within these transfer patterns was striking: Although 
the vast majority of students transferring out of DPS were lower in poverty than DPS itself, in 
three cases students were transferring to districts with higher (albeit slightly higher) poverty rates 
(Adams County 14, Sheridan 2, and Westminster 50). This variation was even more evident when 
examining transfers by race and ethnicity (Figure 2). In the vast majority of cases (9 of 13 cases), 
disproportionately high numbers of White students were transferring to districts with higher 
proportions of Whites compared with Denver. In two cases (transfers to Adams County 14 and 
Mapleton 1), however, disproportionately high numbers of Hispanic students transferred to dis- 
tricts with relatively greater numbers of Hispanic students (although not lower poverty students). 
In one case (Vilas RE-5, the online charter school), Black students were disproportionately likely 
to leave DPS for the school, which had higher numbers of Black (although not lower poverty) 
students (see also Table 4). 

This case begins to illustrate the stratifying nature of choice in the Denver metro area. Although 
DPS serves a largely non-White and low-income student population, it lost many of its White and 
higher income students to surrounding districts through interdistrict choice. In three cases, the 
district lost relatively higher income, higher minority populations to other districts and/or charter 
schools. 


Sender #2: Jefferson County R-1. The second-largest “loser” of students through in- 
terdistrict choice was Jefferson County Public Schools, a predominantly White (75.6%) and 
low-poverty district (24.1% of students eligible for free or reduced-price lunch). Although Jef- 
ferson County lost 2,751 students to interdistrict choice, it also received 3,933 students and was 
therefore a net receiver of 1,182 students. 

Jefferson County’s story is complicated: On average, students leaving Jefferson were slightly 
lower income than the Jefferson County Public Schools average (by 5.4 percentage points). More 
significantly, students were choosing districts that with higher poverty rates which were, on 
average, nearly twice that of Jefferson County. The same patterns were evident with respect to 
race and ethnicity. Students transferring out of Jefferson were on average disproportionately more 
likely to be Hispanic and less likely to be White than Jefferson County as a whole. Likewise, 
on average, students transferring out of Jefferson were choosing districts that have significantly 
lower proportions of White students and higher proportions of Black and Hispanic students. Thus, 
overall, it appears that lower income and non-White students were using interdistrict choice to 
leave Jefferson in favor of school districts with larger proportions of lower income and non-White 
students, again contributing to the between-district stratification effect from interdistrict choice 
observed with DPS above. 

Moreover, a closer look at the variation in transfer patterns illustrates an even more powerful 
stratification effect. In four of the 12 transfer pairs, transfer students leaving Jefferson were 
disproportionately lower income compared with Jefferson’s average student population, and left 
Jefferson for districts that had far higher numbers of low-income students. For example, students 
who transferred from Jefferson to DPS were twice as likely to be eligible for free or reduced-price 
lunch as Jefferson students overall. Further, DPS has a poverty rate more than two and a half 
times that of Jefferson. In each of these four pairs, White students were disproportionately less 
likely to transfer and Black and Hispanic students were disproportionately more likely to transfer. 
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In the other eight transfer pairs, however, higher income students vis-a-vis Jefferson County 
were leaving Jefferson County, and in six of these cases the students transferred to districts with 
significantly lower proportions of low income students compared with Jefferson. Further, in seven 
cases, White students were more likely to transfer out of Jefferson County. 

Thus, as a result of interdistrict choice, Jefferson County—a predominantly White and rel- 
atively affluent district—had been losing its lower income students and students of color to 
districts with larger proportions of lower income students of color, and losing higher income 
White students to districts with even higher concentrations of White and affluent students. 


Sender #3: Adams 12 Five Star Schools. The final sender in this analysis was Adams 
12 Five Star Schools, which lost 2,644 students but, like Jefferson, was a net receiver of 1,390 
students overall. Like Jefferson, Adams 12 Five Star is predominantly White (61.7%) and low 
poverty (29.2% of students were eligible for free and reduced-price lunch). The district is not 
adjacent to DPS, but is “buffered” by three districts north of DPS (Westminster 50, Mapleton i. 
and Adams County 14). Adams sent the plurality of its interdistrict transfer students (48%) to 
charter schools. In 2006-07, 973 students transferred to the Charter School Institute (a collection 
of charter schools) and another 314 transferred to Vilas RE-5, the virtual online charter school. 

On average, students exiting Adams were less likely to be White, slightly less likely to be 
Black, and more likely to be Hispanic vis-a-vis Adams students. Further, on average, students 
leaving Adams chose to attend districts that were significantly higher in poverty; lower in White 
student populations; and, accordingly, slightly higher in Black and Hispanic student populations 
than Adams. 

The disaggregated trends were more complicated and tend to go in two analytic (and ge- 
ographic) directions: In one direction, Whiter and more affluent districts from the north and 
west (i.e., Boulder Valley and Jefferson County) were siphoning off disproportionate numbers 
of White and higher income students from Adams. At the same time, lower income students of 
color were exiting Adams to attend more diverse and lower income districts to the South, such as 
DPS and Mapleton. Taken together, these patterns again highlight how interdistrict school choice 
is exacerbating between-district stratification and segregation. 


The Receivers: Stratification from the Winner's Point of View 


The top three “receivers” of students in interdistrict choice, ranked by absolute number of students 
transferring out, were the same as the top three senders although in a different order: (a) DPS, 
which took in 4,491 students; (b) Adams 12 Five Star Schools, which took in 4,034 students; and 
(c) Jefferson County, which gained 3,933 students. Together, these three top receivers received 
55% of all transfers from the region’s 15 districts. As noted previously, Jefferson County and 
Adams 12 Five Star Schools were both net receivers, whereas DPS was a net sender. 





Receiver #1: DPS. DPS, a high-poverty (66.9% free and reduced-price lunch) and high- 
minority (75.3% Hispanic and Black) district, would not, under the existing interdistrict choice 
literature, be predicted to be a large “receiver” of transfers through interdistrict choice. (To be 
sure, DPS is indeed a “net loser” of 1,828 students to other districts.) 
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At first glance, DPS’s transfer patterns appear to run counter to the patterns documented in 
the first half of this article. On average, students who transferred into DPS came from districts 
with free and reduced-price lunch rates less than half that of DPS. In addition, transfers to DPS 
came from districts with twice the proportion of White students as DPS as well as significantly 
lower proportions of Black and Hispanic students (Table 5). Further, as a whole, the transfers-in 
themselves were significantly Jess poor (12.8 percentage points), more White (8.6 percentage 
points), and Jess Hispanic (10.0 percentage points) than the DPS student population. They were 
also more likely to be Black than DPS students (4.7 percentage points). Yet, compared with their 
home (or sending) districts, the students who transferred into DPS were, on average, more than 
twice as likely to be in poverty; less than half as likely to be White; and, accordingly, significantly 
more likely to be Black and Hispanic (Table 5). 

Disaggregating these data helps to make sense of these trends. Looking closely at the three 
largest senders of students into DPS (which account for 68% of all students into DPS), it becomes 
clear that two of the districts—Cherry Creek to the southeast and Jefferson County to the west 
(both of which are majority low-poverty and White)—sent disproportionately high numbers of 
low-income and non-White students into DPS. The other largest sender, Adams-Arapahoe, is a 
largely high-poverty and non-White district that lost students who were slightly lower in poverty 
levels to DPS and who were slightly more White on average. 

Thus, on balance, DPS is contributing to between-district stratification and segregation by 
siphoning non-White and lower income students from predominantly White districts, causing 
those districts to become even more affluent and White (especially in the context of the transfer 
trends documented in the rest of this article.) Although DPS gained some lower poverty and 
White students through interdistrict choice, on balance it lost more Whites and higher income 
students than it gained. 


Receiver #2: Adams 12 Five Star Schools. The second largest receiver of interdistrict 
transfers in the Denver metro area was Adams 12 Five Star Schools, which was also a net receiver 
of 1,390 students. What is particularly interesting about this district is that it received relatively 
small numbers of transfers from a large number of districts. The maximum number of transfers 
from one district to Adams 12 Five Star Schools was 388, and the ten top senders combined 
only comprised 51.1% of all the transfers into Adams. As noted previously, Adams 12 Five Star 
Schools enrolls 37,341 students and is just under two thirds White and nearly one third Hispanic, 
with a small minority of Black students. Slightly less than 30% of the student population is 
eligible for free- or reduced-price lunch. 

Striking patterns emerge from an analysis of the transfers into Adams: On average, transfers 
to Adams were significantly less likely to be poor than students in their home districts (28.8 
percentage points lower in free and reduced-price lunch eligibility) and were leaving districts 
with substantially higher poverty levels than Adams (by 18.3 percentage points.) Transfers were 
also much more likely to be White compared with their home districts and came from districts 
that, on average, had higher proportions of Black and Hispanic students than Adams. Overall, 
Adams was siphoning off more affluent and disproportionately White students from less affluent 
and more diverse districts through interdistrict choice. 

A closer look at the nuances of the transfer patterns reveals that, even in those cases in which 
students transferred to Adams from districts with lower rates of poverty and lower portions of 
Hispanics, these transfers did not have a significant impact on stratification, as these transfers 
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consisted largely of transfers of students from affluent and White districts to a slightly less affluent 
and White district. : 


Receiver #3: Jefferson County R-1. Jefferson County was the third largest receiver of 
students and a net receiver, which received 3,933 students through choice, and “netted” 1,182 
students through the open enrollment program. As noted previously, Jefferson County Public 
Schools is a predominantly White (75.6%) and low-poverty (24.1%) district. 

On average, as with Adams, transfers into Jefferson were less than half as likely to be poor as 
students in their home districts and were transferring from districts with poverty rates more than 
double that of Jefferson. Transfers were much more likely to be White compared to their sending 
districts (32.3 percentage points higher) and were less likely to be Black or Hispanic compared 
to their sending districts. Further, transfers were relocating to Jefferson from districts that were 
less White, more Black, and more Hispanic. 

In examining each sending-receiving pair, it is clear that patterns contrary to those noted 
previously are exhibited by three districts in particular: Douglas County, Littleton, and Boulder 
Valley. In each case, the analysis shows that these predominantly White and affluent districts 
are sending larger proportions of higher income and White students, and lower proportions of 
Hispanic and Black students, to Jefferson County. Yet in each of these cases, as with Adams, 
the effect on overall between-district stratification is quite small as these transfers simply consist 
of White and relatively affluent students transferring to yet another White and relatively affluent 
district. Overall, Jefferson County—like Adams—was siphoning off White and more affluent 
students from higher poverty and predominantly less- White districts. 


DISCUSSION AND CONCLUSIONS 


These analyses underscore the importance of examining transfers within their metropolitan con- 
text. As the findings illustrate, previous studies of interdistrict choice focusing on average or 
aggregate data in terms of race, ethnicity and income have masked complex and striking transfer 
patterns. 

The most important finding from this analysis is that interdistrict choice appears to contribute 
to regional, between-district stratification. The most significant stratification occurs with respect 
to socioeconomic status: Across the region, higher income students were far more likely to take 
advantage of interdistrict choice and to transfer to higher income school districts. Although the 
lack of school-level data makes it difficult to determine what types of schools these students were 
transferring to, it seems fair to assert that they were transferring into lower poverty schools given 
the lower overall poverty rates of the chosen districts. 

Stratification effects were also found in terms of race and ethnicity: On the whole, White 
students were more likely to transfer out of racially diverse districts into districts with higher 
proportions of White students. Although information is not available on the specific schools that 
students chose, it can again be presumed, given the large portions of White students in the district 
that these students were opting into, that the schools chosen were also predominantly White. 

Several possible explanations for these findings in the context of Colorado’s interdistrict 
choice policy may be offered. First, the level and character of student participation in choice 
programs is closely related to the structure of the choice program itself (Holme & Wells, 2008). 
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Choice programs that have fewer supports for students from families with low levels of income 
or education (such as information for students and free transportation) tend, predictably, to 
have lower participation rates among that demographic. Colorado’s choice program is entirely 
deregulated: There are no requirements that parents be informed about choice options, and no 
requirements to provide students with transportation. As such, it is perhaps unsurprising that 
lower income students are less likely less likely to participate in Colorado’s interdistrict choice 
program. 

Second, the low participation rates of lower income students of color may also be traced to 
possible district filtering of students. Although there has been no direct evidence of this filtering 
within this research, the Colorado law’s language is vague and permits districts discretion in how 
many students and what type of student they admit, as stated earlier. In addition, under the law, 
school districts are permitted to deny admissions to special education students. Although the law 
does mandate that districts “consider” adopting provisions granting preferences to students in 
low-performing schools, districts are not required to do so (Colorado Public Schools of Choice 
Act, 1990). 

This analysis also reveals a somewhat surprising pattern that also contributes to regional, 
between-district stratification among school districts. In some cases, lower income students and 
students of color used interdistrict choice to flee from predominantly White and affluent districts 
to districts in which there were higher proportions of students of their own backgrounds. This 
was primarily true of students transferring into DPS from the largely White and relatively affluent 
suburbs of Jefferson County, Cherry Creek, and Adams 12 Five Star Schools. In each of these 
cases the “out transfers” were largely students of color and more likely to be eligible for free and 
reduced-price lunch than students in the district that they left. It is important to note, however, that 
although these students were relatively more likely to be low income compared to most students 
in their home district, they were still less likely to be low income than most students in the DPS. 
It should also be emphasized that this phenomenon accounts for a relatively small proportion of 
overall transfer patterns. For example, of the 4,491 student transfers into DPS, only one fourth 
(1,193) were more likely to be free and reduced-price lunch or non-White than the districts from 
which they transferred and were transferring to DPS from more affluent and more White districts. 

There are a number of possible explanations for this (minor) trend. First, DPS operates a 
number of magnet schools that are open to interdistrict transfers, which offer relatively more 
diverse learning environments and which may be attractive for that reason (see also Henig, 1996). 
DPS also houses a number of charter schools, which are also open to interdistrict transfers. Many 
of these charters are designed to serve students who are at risk of dropping out or who had 
already dropped out of their traditional high school. As discussed previously, a large number 
of low-income students of color transferred to Vilas-RES, the virtual online charter school for 
students at risk of dropping out, which illustrates the popularity of these types of programs. 

The findings of this study should concern policymakers for several reasons. The first is the 
apparent contribution of interdistrict choice to the already growing concentration of poverty in 
high-poverty school districts (Clotfelter, 2004). Concentrated poverty is associated with a host of 
educational challenges including high mobility and absenteeism, each of which tend to disrupt 
daily school operations (Kahlenberg, 2001). High-poverty schools, furthermore, consistently have 
difficulty attracting and retaining qualified teachers (Lankford, Loeb, & Wyckoff, 2002). The 
concentration of poverty in schools also appears to influence achievement: Research has found 
that low-income students in high-poverty schools tend to exhibit lower achievement and make 
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fewer gains in achievement than comparable students attending schools with larger proportions of 
middle class students (Kahlenberg, 2001; G. Orfield & Lee, 2005; Rumberger & Palardy, 2005). 

The second reason for concern is the evidence from this study about the contribution of 
interdistrict choice to the growing isolation by race and ethnicity between districts. Research has 
shown that interracial contact in schools generates a host of short- and longer term academic and 
social benefits (Wells & Crain, 1994). ‘ 

These findings suggest that interdistrict school choice programs under NCLB will require 
significant controls, supports, and oversights to encourage the participation of the least advantaged 
students. The most successful examples of equity-minded interdistrict choice policies come from 
long-standing voluntary interdistrict desegregation programs, which ensure that students are 
provided information, transportation, and support in their chosen schools (Holme & Wells, 
2008.) A number of these desegregation programs also include strategies designed to attract more 
affluent students into higher poverty districts through interdistrict magnet schools (Eaton, 2007). 
While the U.S. Supreme Court’s decision in Parents Involved will most likely prohibit NCLB 
regulations from including provisions to ensure racial and ethnic balance in any transfer program, 
the findings in this study illustrate a need to carefully control for, and monitor, the poverty status 
of transfer students. 

This study was unable to address the other major assumption undergirding interdistrict choice 
policies: the notion that competitive market pressures will compel school districts to improve. 
Certainly, districts experienced the pressures of choice in this context, yet it is unclear what 
types of pressures were felt, or on what measures or indicators parents were choosing. The 
overwhelmingly consistent findings across these analyses, however, suggest that at least on 
some level, race and social class do matter to parents. Additional research incorporating school- 
level data, however, is necessary to fully understand how, and to what degree, districts may be 
responding to market demands created by interdistrict choice policies. 
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This study investigates how much the racial composition of communities influences the private school 
enrollment rates of members of different racial groups. Some scholars argue that private school 
enrollment contributes to racial segregation in public schools because White families attempt to 
enhance the social status of their children by leaving public schools serving communities with higher 
percentages of children who are Black. A second group of scholars argue that private school enrollment 
is primarily based on nonracial factors. A third, related perspective argues that race is of diminishing 
importance in driving behaviors such as school choice. This study explores these perspectives using 
1990 and 2000 Public Use Micro Data Samples to estimate private school enrollment rates by student 
race and community racial composition. Findings indicate that private school enrollment rates among 
Asian, Black, and Hispanic students do not fluctuate much with community racial composition, By 
contrast, private school enrollment rates among White families are strongly and positively correlated 
with the percentage of children in their communities who are Black—even after holding constant 
a series of individual and community-level factors that may account for this trend. Moreover, the 
association between race and choice has changed little between 1990 and 2000. 


Recent scholarship on school choice has consistently revealed that racial segregation across 
schools is higher than that across their corresponding school attendance zones. For example, 
Saporito and Sohoni (2006) find that more than three fourths of the 22 largest school districts 
in the United States have greater White/Black and White/Hispanic segregation among student 
populations enrolled in local public schools than exists among student populations living in 
school attendance zones. Similarly, studies of public school choice in Philadelphia (Saporito, 
2003) and Durham (Bifulco, Ladd, & Ross, 2008) show that racial segregation between White 
and non-White students is higher across neighborhood schools than across their corresponding 
catchment areas. This body of recent scholarship raises an intriguing question: Why is racial 
segregation higher among students enrolled in schools compared with students who live in their 
corresponding attendance zones? 
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This study develops and empirically assesses a conceptual framework that explains the dis- 
crepancy between racial segregation across schools and their corresponding attendance zones. 
Specifically, I examine two theoretical perspectives that explain how student choices for non- 
neighborhood schools may contribute to racial segregation in local public schools. The first 
theoretical argument suggests that many members of historically advantaged racial groups (1.e., 
Whites in the United States) attempt to maintain their social status by avoiding places occupied 
by historically disadvantaged groups. This perspective is commonly called “out-group avoid- 
ance” in the sociological literature. A competing perspective is that members of all racial groups 
seek social spaces occupied by members of their own group—a theory best summarized by the 
phrase “same-race affinities.” Both of these perspectives imply that large-scale patterns of racial 
segregation emerge from the individual] choices of students. 

To explore which theoretical perspective best anticipates race-based patterns of private school 
choice, I use U.S. Federal Census Data from the 1990 and 2000 Public Use Micro Data Sample 
(PUMS) to examine the private schoo] enrollment patterns of individual Asian, Black, Hispanic, 
and White students. These data allow me to determine if private school enrollment rates of students 
from different racial groups are associated with the racial composition of the communities in which 
they live. A primary issue that motivates these empirical analyses is whether White children avoid 
enrolling in public schools when they live in communities with higher proportions of non-White 
children or if the members of all racial groups are more apt to avoid public schools as shares of 
their own racial group across communities declines. Although patterns of out-group avoidance 
and same-race affinities are distinct, each could lead to greater racial segregation in local, public 
schools than would otherwise exist if all students enrolled in their currently assigned public 
school. 

The main contribution of this article is to empirically substantiate which theoretical perspective 
best anticipates patterns of private school choice found in recent microcensus data describing 
students living in every corner of the United States. Based on existing theoretical literature 
on racial dynamics (Blumer, 1958; Bobo, 1999) and case studies of school choice within a 
single city (Bifulco et al., 2008; Hastings, Kane, & Staiger, 2006; Lankford & Wyckoff, 2006; 
Saporito, 2003), I anticipate that private school enrollment rates are best characterized by the out- 
group avoidance model of racial dynamics. Still, few studies explore choice patterns of multiple 
racial groups using data that are representz ive of all U.S. students. Although these analyses 
are motivated primarily by understanding the particular choice dynamics that may contribute to 
racial segregation in public schools, my research also speaks to policy debates regarding “school 
choice.” Findings produced in this work suggest that, in the absence of regulations that prevent 
particular schools from becoming racially imbalanced, school choice programs are unlikely 
to reduce racial segregation below that which would exist in neighborhood-only assignment 
policies. 


RELATED LITERATURE ON SCHOOL SEGREGATION 


In most school districts, students are assigned to a local, public school—but many students attend 
schools other than the ones serving the catchment areas in which they live. As a result of these 
individual choices, the racial composition of local public schools are not the same as the racial 
composition of the catchment areas that each school serves. Moreover, racial segregation across 
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schools is greater than racial segregation across the areas that are served by these neighborhoods 
(Bifulco et al., 2008; Saporito, 2003; Yancey & Saporito, 1995). 

The most extensive of these studies collected maps of school attendance zones for the 22 largest 
school districts in the United States (Saporito & Sohoni, 2006). For each school district, school 
attendance boundary maps were collected and combined with U.S. Census Data to determine the 
number of students who lived in a school catchment area by age and race. Once the number of 
students living in each catchment area was determined, racial segregation between White and 
Black students and between White and Hispanic students was calculated (using the Index of 
Dissimilarity) to determine the degree to which White and non-White students were unevenly 
distributed across school zones within a district. Racial segregation of students across school 
attendance zones were then compared with levels of segregation across the set of schools that 
served a fixed catchment area. For example, the Detroit School District contains 154 elementary 
schools that have an attendance boundary; racial segregation between White and Black children 
across school catchment areas is 10% lower than White/Black segregation across the 154 schools 
that serve these boundaries. Results are consistent for most other school districts and at the 
elementary, middle and high school grade levels. 

Similar studies have been conducted in countries such as England, Wales, and the Netherlands 
(Burgess & Wilson, 2005; Burgess, Wilson, & Lupton, 2005; Johnson, Burgess, Wilson, & Harris, 
2006; Karsten, Ledoux, Roeleveld, Felix, & Elshof, 2003; Willms & Echols, 1992). For example, 
Burgess et al. (2005) used dissimilarity to measure White/Black segregation across schools and 
neighborhoods within Local Education Authorities in the United Kingdom.! They find that across 
schools and neighborhoods within Local Education Authorities, “children are more segregated 
in school than in their neighborhood” (p. 1052). Moreover, most of these studies find greater 
segregation across schools than residential areas for most ethnic group pairings. For example, 
there are higher levels of segregation across schools than across neighborhoods between White 
students and students from the following ethnic groups: Africans, Black Caribbeans, Chinese, 
Indians, and Pakistanis. 

A second body of literature that is related to comparisons of racial segregation between 
schools and their catchment area explores historical shifts in racial segregation across U.S. 
schools and U.S. neighborhoods. These studies, based on the decennial census and the Common 
Core of Data, indicate that Black-White racial segregation in residential areas has decreased over 
the past several decades, while segregation across schools has increased (Frankenburg, Lee, & 
Orfield, 2003; Reardon & Yun, 2003). A number of scholars argue that the rise in public school 
segregation might be explained by the movement of children from the public to the private school 
sector (Clotfelter, 2004; Hastings et al., 2006). 

Although interesting empirically, these results also present an interesting theoretical puzzle: 
What racial dynamics explain the discrepancy between racial segregation across schools and 
school catchment area racial segregation? Some scholars argue that families seek schools in 
which most students are like themselves racially (Clark, 1991, 1992; Patterson, 1997; Thern- 
strom & Thernstrom, 1997). From this view, racial segregation emerges from a benign form of 
ethnocentrism in which people congregate in social settings occupied by others of “their own 
kind” because they have “shared interests, similar sensibilities or tastes, a common culture, sense 


'The “neighborhoods” in these studies are not the same as school catchment areas. 
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of mutual acceptance, and in-group solidarity” (Fossett, 2006, p. 187). Indeed, studies have shown 
that many members of the general public assume that most individuals want to surround them- 
selves with others who are like themselves culturally. Work by Bonilla-Silva (2006) and Clark 
(1991, 1992) presents persuasive empirical evidence that racial segregation patterns may result 
from same-race preferences. 

In stark contrast to the practice of assuming that ethnic processes and concepts apply to racial 
categories, the group position theory of race developed by Blumer (1958) and elaborated by 
Bobo (1999) and Bobo and Tuan (2006) maintained that race is a socially constructed system 
of inequality in which some racial groups actively seek to maintain a privileged social position 
in relation to disadvantaged racial groups. From this view, segregation in schools and residential 
areas is itself an instantiation of dominant group privilege. In the context of the contemporary 
United States, one way Whites try to maintain their superordinate status is by avoiding social 
spaces occupied by members of subordinate racial groups—particularly social spaces inhabited 
by African Americans. Of importance, the group position theory of race suggests that some 
members of less privileged racial groups seek greater social inclusion through integration, al- 
though such opportunities remain elusive because of the racial preferences of White people and 
their penchant for engaging in various forms of discrimination (Massey & Denton, 1993; Yinger, 
1995). 

To assess which of these theoretical models best anticipates the school enrollment patterns 
of students, it is necessary to compare the school choices of children from different groups. A 
variety of studies have explored private school (or magnet school) enrollment patterns of White 
and non-White students to assess this issue. Among the most important of these studies was 
Lankford and Wyckoff’s (2006) investigation of private school enrollment patterns of White 
students. These authors conducted statistical analyses to determine whether an association ex- 
isted between the probability that a White child was enrolled in private school and the racial 
composition of the children in each child’s school catchment area. The research was based on 
eight metropolitan areas in upstate New York. These scholars find that White families were 
more likely to attend private schools as the percentage of non-White students in a neighbor- 
hood school catchment area increases, even after controlling for a variety of student and school 
characteristics that might reasonably account for White students’ choice behavior. Related work 
by Fairlie and Resch (2002) also shows that White students were increasingly likely to attend 
private school as the percentage of minority students in their county (or metropolitan area) 
increased. 

These studies make an important contribution to our understanding of how White families make 
choices as shares of non-White children in an area increases. Still, there are several limitations 
with this work. The most critical limitation is the lack of information describing the school 
choices of Asian, Black and Latino students. Without examining the choice of students in these 
groups, it is not possible to gain insight into whether choice patterns conform with the competing 
theoretical expectations of out-group avoidance or same-race affinities. A second limitation with 
the work of Fairlie and Resch (2002) is that the social and economic characteristics that may 
influence school enrollment often describe metropolitan areas. Metropolitan areas are large and 
internally diverse and such data might not accurately capture the local context that influences the 
choices of individual students. Although Lankford and Wyckoff (2006) addressed this limitation 
by exploring units of analyses that are ideal (i.e., school catchment areas), they do so for a 
relatively small region of the country (eight metropolitan areas in upstate New York) where well 
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over 80% of the population is White (for the region as a whole and for each of the metropolitan 
areas included in their study). ‘ 

This article builds on the existing literate by examining the specific private school enrollment 
patterns of White and non-White children to identify whether (a) White children are more apt to 
enroll in private school as the proportion of non-White children in their neighborhoods increases, 
or (b) White and non-White children are more apt to enroll in private school as the proportion 
of children of their own race diminishes. I also explore whether economic differences between 
racial groups account for some (or all) of the differences in private school enrollment patterns 
among different racial groups and I explore whether private school enrollment patterns among 
various racial groups have changed between 1990 and 2000. 


DATA AND METHODS 


To examine private school attendance patterns I assemble data from the Census Bureau’s 
1990 and 2000 PUMS file for the entire United States.2 The microdata contain records for 
5% of all U.S. households. This enables me to compile a nationally representative sample of 
450,689 children (ages 14 to 18 who are enrolled in high school) in 1990 and 530,024 in 
2000. These children represent the full spectrum of racial and economic diversity in the United 
States.> 


Dependent Variable 


The PUMS data ask respondents whether they are enrolled in school and, if so, whether it is 
public or private. This enables me to construct a binary dependent variable specifying whether 
a student is enrolled in private school (assigned a value of 1) or public school (equal to 0).4 
Separate estimates of private school enrollment are made for four mutually exclusive racial 
groups: (a) White non-Hispanics, (b) Black non-Hispanics, (c) Asian non-Hispanics, and (d) 
Hispanics. Children classified as Native Americans, “others,” or multiracial were excluded from 
these analyses because of small sample sizes or lack of comparability in these racial categories 
between 1990 and 2000. Estimating separate models for each combination of race simplifies the 
analyses by eliminating the need to create a large number of interaction terms between a student’s 


21 extracted data from the Integrated Public Use Micro Data Sample (IPUMS; Ruggles et al., 2008) which allowed 
me to use variables that are the same in 1990 and 2000. 

3Roughly 8% of the children in the PUMS data were not the children or stepchildren of the householder and are 
excluded from these analyses. For these students, it is not clear whether the head-of-householder was the primary decision 
maker in determining whether a child was enrolled in private school. More important, the PUMS data do not provide 
any information describing how much economic support children in nonparent households receive from their parents and 
how much they receive from householders. The lack of information describing the level of economic support provided by 
absentee parents makes it impossible to include adequate socioeconomic controls in statistical analyses of private school 
enrollment. These students are about half as likely as related children to attend private school. 

4To simplify the presentation of the findings in the main body of the text, I report the findings for high-school-aged 
children. Analyses of middle- and elementary-school-aged children produces results that are substantively the same as 
those shown throughout this article. Results for middle- and elementary-school-aged children are available upon request. 
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race, community racial composition, and the remaining independent variables included in the 
analyses. 


Student- and Household-Level Measures 


Table 1 shows the descriptive statistics of student and household characteristics by racial group 
for 2000 data (1990 data are available upon request); these variables will be incorporated as 
controls in statistical analyses presented later in this article. Among the most important of these 
is the Hauser and Warren socioeconomic index of occupational prestige available in the IPUMS 
(Ruggles et al., 2008). This measure of occupational status is based on the earnings and educa- 
tional attainment associated with each occupational category in the 1990 occupational scheme 
(Hauser & Warren, 1997). I use the highest socioeconomic index of the householder or their 
spouse. Similarly, I use the highest level of education obtained by either the householder or 
their spouse. Total household income was the final measure of family economic characteristics 
that might influence private school enrollment. Finally, current employment status of the house- 
holder was used to describe the family’s immediate economic circumstances (where unemployed/ 
NILE:=.1). 

Three variables were used to estimate a household’s wealth: whether the householder(s) rented 
(where rented = 1); whether the householder(s) owned their home “free and clear” without a 
mortgage (where owned home = 1); and the value of owner-occupied housing units (in 2000 
dollars). I also controlled for whether a householder was married (where married = 1). The 
number of children in a household was also calculated. This variable measures the strain on 
household resources. 


Community-Level Measures 


In addition to student and household characteristics, I also measured several important char- 
acteristics of the PUMS areas (hereafter called “communities”) in which children lived. 
Community-level summary statistics are shown in Table 2. To create community-level vari- 
ables, I aggregate child and family characteristics for 2,070 communities in 2000 (and 
1,726 communities in 1990). To measure a community’s racial composition, I calculated 
the percentage of Black, school-aged children (aged 5-18) in each area using complete- 
count census data (complete-count census data eliminates error that would result from sample 
information). 

To capture the economic conditions of an area, I measure the percentage of persons (older than 
25) with at least a college degree and the percentage who are unemployed. Median household 
income measures average community economic conditions. I also measure economic inequality 
using the Gini Index. Gini calculations are based on the annual household income of all house- 
holders between the ages of 25 and 66. To account for the potential influence of urban and rural 
location, I calculated the number of students per square mile within each community. Commu- 


nities with higher densities of children (i.e., those that are typically within metropolitan areas) 
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TABLE 1 
Descriptive Statistics of Student and Household Characteristics Among Students Enrolled in High School 
by Race, 2000 Data 


- DR erie rd ee ee ee aa ee 








Children of all Races* Asian Black Hispanic White 
Variable M SD M SD M,. SD M SD M SD 
Dependent variable 
% in private school 8.3 — 8.4 — 42 — 5.3 _— 9.8 — 
Student and 
household 
characteristics 
% students who are 50.7 —- 51.6 — 49.3 — 50.6 — 50.9 — 
male 
Age 16.3 it] 16.1 Let 16.2 Leal 16.1 el 16.3 1.0 
% householders TAee — 81.4 — 42.6 —_ 67.7 — Wile? — 
married 
Family size 4.2 1.6 4.8 1.9 4.3 1.8 4.9 2.0 Al eS 
Log household 10.7 ies 10.7 L-5 10.2 te 10.4 eS 10.9 1.0 
income? 
% own home free and = 12.5 — 7.6 — 9.8 — 10.7 — 13.4 — 
clear 
% rent dwelling 24.1 — 34.0 — 44.5 as 42.0 — 15.8 — 
Value of house? 119,027 145,787 159,488 191,856 55,354 84,131 73,951 107,143 139,275 153,469 
% householder(s) less 10.5 — 14.5 — 14.7 — 36.7 — 4.1 — 
than HS 
% householder(s) HS 27.9 — 20.0 — 36.0 — 27.6 — 26.7 — 
% householder(s) 32.8 — 22.6 — 33h — 24.0 — 34.9 — 
some college 
% householder(s) 28.8 — 43.0 a 15.6 — ili ley — 34.2 
college 
Householder(s) 3,803 1,716 317.89) 990 SaB\08 4 1/24 2.007 S25 a aoe e635 
socioeconomic 
status 
N students 537,192 20,724 71,386 74,020 363,894 





Note. HS = high school. 
4Children of all races include students who are not Asian, Black, Hispanic, or White. >Income amounts in 1999 dollars. 


come close to resembling the size of two or three school feeder areas.° By contrast, in rural areas 
communities can cover vast areas.° 


>Most of these dense communities encompass the territory of roughly two or three urban high schools. For example, 
the Los Angeles school district has 79 high school catchment areas that are contained within 34 communities; similarly, 
Philadelphia has 22 neighborhood highs schools and 11 communities. In this study, one third of the communities are 
less than 31 square miles and two thirds of the areas are less than 85 square miles. There are only six communities that 
are between 1,000 and 1,200 square miles and these areas contain large towns. An example is the southern tip of Texas, 
which contains the town of Brownsville. 

®All variables measuring nonracial factors are generated by aggregating individual-level sample data to PUMS areas. 
Because these aggregations are generated using sample information, there is an error term associated with each variable 
for every community. Still, the errors are very small given that the typical number of observations used to compute any 
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TABLE 2 
Descriptive Statistics of Community Characteristics Among Students Enrolled in High School by Race, 
2000 Data 
Children of All Races* Asian Black Hispanic White 
Variable M SD M SD M SD M SD M SD 
% students in 13.6 Loe 10.6 13:85 45 27.0 10.9 14.0 9.0 13.0 
community who are 
Black 
% students in 66.4 28.0 ATS) Vee Ti AmiecA2 Sit 26.600 MS S4e yee 2 eNO O 19.6 
community who are 
White 
% students in 20.0 231 42.1 27.6 16.1 LSS 50!7 = 927-4 13.1 158, 
community who are 
NBM 
(A) 14.0 8.0 13 6.6 18.5 9.6 14.3 7.8 sll 73 
Dw/b 48.0 137 42.2 1390 A9'5 15.0 45.8 14.8 48.4 13s 
% not citizens 4.0 4.9 8.4 6.0 4.1 4.8 92 6.3 2.8 3.6 
School-aged children 23.0 59:2 48.1 [OT S62) D9 99:8 [ieee 292 
per square mile 
% college 23.8 12.4 30.7 VAS te W2i1cGiky GieleSiangy220:3 11.8 245 12.4 
% unemployed a2 1.5 3.4 iS 4.0 1.8 4.2 eo 2.8 it 
Median household 43,156 13,936 51,180 16,312 37,733 12,155 40,363 13,146 44,443 13,852 
income 
Gini coefficient of 39.6 4.0 SOS, 4.1 41.9 Arye Al 4.1 38.8 3.6 
income inequality 
N communities 2,070 1,846 1,981 2,026 2,069 


i 


Note. (H), Dw/», and Gini were multiplied by 100 for purposes of presentation. Non-Black Minority (NMB) = xx; 
(H) = Thiel’s entropy index segregation across census tracts within a community. 
4Children of all races also include students who are not Asian, Black, Hispanic, or White. 


Because of the likely separation of racial groups within communities, I calculated internal 
residential racial segregation across census tracts within each community using Theil’s (Theil 
& Finezza, 1971) entropy index of segregation (abbreviated as H).’ In this study, H is based 





community-level characteristic is large (at least 225 observations for student characteristics aggregated to communities 
and at least 635 observations for head-of-household characteristics). For example, the standard errors of community-level 
Gini coefficients range from .004 to .017 and the largest standard error for the percent of children who were not citizens 
was 2.2 percentage points. 

7To calculate H, it is necessary to first calculate “entropy” (E): 


r 

1 

E; = do ilny 
ral 


where J, is the proportion of a community’s population comprised of racial group r. Entropy must be calculated for the 
community as a whole (E) and for each tract within a community (Ej) Values of H are then calculated with 


Where T is the total population of the community and t; is the population of the tract. 
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on six racial categories in the census (i.e., people who were Hispanic or non-Hispanic Asian, 
Black, Native American, White, or other) and essentially measures the average racial diver- 
sity of census tracts within a community compared with the overall racial diversity of the 
entire community. Values of H range from zero to one, where zero represents a state of per- 
fect integration (which results if every tract within a community has the same racial diversity 
as that of the entire community) and one represents perfect segregation (i.e., every tract is 
mono-racial). 

To examine the particular effect of intracommunity segregation between Black and White 
children, I also included the more traditional measure of Dissimilarity.® In this study, Dissimilarity 
assesses the degree to which Black and White children are evenly distributed across census tracts 
within communities. Measuring the spatial heterogeneity of racial groups within communities is 
important because, for example, White children who live in a community with high segregation 
are, by definition, likely to live in a subsection of the community with high percentages of other 
White children. Intercommunity segregation could influence a student’s likelihood of enrolling 
in a private school independent of the racial composition of the community as whole. The 
summary statistics of the 2000 communities are shown in Table 2 (1990 data are available upon 
request). 

Ideally, I would like to have included school quality data to assess parents’ choices about school 
selection, but the microcensus data do not allow me to integrate school-level information into my 
analyses. Although I can determine the extent to which private school enrollment is correlated 
with a community’s racial composition, I cannot determine whether such a correlation would 
diminish (or potentially disappear) if I could incorporate a complete set of variables that parents 
might consider when making residential and school choices. Three of these factors are school 
quality, community (or school) safety, and the religious denomination of students’ families. In 
particular, the decennial census does not collect this information. Similarly, it would be helpful to 
control for the percentage of a community’s population that is of a particular denomination (e.g., 
the percentage of people who are Catholic) but available data describing religious populations 
cannot be integrated with PUMAs. 

Despite the unavailability of data that might be able to strengthen my ability to make robust 
causal inferences regarding the association between private school enrollment and the racial 
composition of local communities, my data will be able to explore how much patterns of private 
school enrollment are correlated with the racial composition of communities in which children 
live. As Brunner, Imazeki, and Ross (2006) noted, correlations between student race and the 
racial composition of local communities allow me to explore whether private school enrollment 
patterns might contribute to racial segregation of local public schools above and beyond that 
which occurs because of residential patterns—even if such correlations are not evidence that 
the racial composition of communities influences families to enroll their children in private 
schools. 


8Dissimilarity reflects the degree to which White and Black children are evenly distributed across census tracts within 
a PUMS area. For details of calculation methods of both Dissimilarity and H, see Reardon and Firebaugh (2002). In this 
article, H and D are calculated using tract-level population counts from the U.S. Census Bureau’s Summary File 1 data. 
Because complete-count data are used to calculate racial segregation across tracts within PUMA areas, H and D have no 
error term. 
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Statistical Modeling Procedures 


Given that the dependent variable is binary (where enrollment in private school equals one) 
and children are nested within communities, I use two-level random intercept logistic regression 
models to estimate the probability that a student is enrolled in private school. I use Stata’s xtlogit 
command to fit random intercept models; this procedure uses numerical integration based on 
adaptive Gauss-Hermite quadrature. I use 12 quadrature points as a larger number has virtually 
no impact on the regression coefficients. The two-level models address the assumption that 
school enrollment among students within the same communities are independent by including 
community-specific random intercepts in the regression models. Because the random intercept 
model corrects for potential intra-cluster correlations of students’ characteristics within each 
community, it produces more efficient parameter estimates and reduces bias in the standard errors 
(Guo & Zhao, 2000; Wong & Mason, 1985). 

If yi; is a binary dependent variable for student i in community j, x represents the values of 
independent variables, and u; is a random intercept for each community, then the log odds that a 
student attends private school Pj; = Pr(yi = 1) is 


“— = oy + Bixi + Uj 
os Pi 





| 
O87 


For ease of interpretation, I graph estimated probabilities of private school enrollment across 
communities that vary in the percentage of their population of a given racial group. Because 
each model includes a random intercept for each community, producing marginal probabilities 
necessitates integrating out the random intercept (i.e., uj): 


Pigs De | e(Bo + Bixij + Uj) 
air ie e(Bo + Bi xi + Uj) 


The predicted probabilities shown in this study represent the effect of community racial composi- 
tion after accounting for other community characteristics that may influence the probability that 
a child attends private school. The estimated probabilities depicted in the graphs hold constant 
all control variables at the means of all children at the high school level (see the “total” columns 
of Tables 1 and 2). Values for dummy variables are set at either zero or one.? 


EMPIRICAL RESULTS 


Before producing and interpreting multivariate regression models, I begin by examining the 
proportion of students who are enrolled in private school by race and the percentage of students 
in a community who are Black. Figure 1 shows that White families are more likely to avoid 
public schools as the proportion of children in their communities who are Black increases. In 
2000, roughly 8% of White children are enrolled in private school if they live in a community 





°The following categorical variables are set at these values: male = 0; married = 1; owns home free and clear = 0, 
rents = 1; parent has some college = 1 (whereas the other educational categories = 0). 
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in which 0 to 10% of the children are Black, yet more than 40% of White children are enrolled 
in private school if they live in a community in which between 71 and 80% of the children are 
Black. The trend for 1990 is similar to that for 2000 (not shown but available upon request). 

Although private school enrollment rates among White students increase with increasing 
shares of Black students in the community, changes in private school enrollment rates among 
non-White students are far less dramatic. Among Black students, private school enrollment rates 
fluctuate between 3 and 6 percentage points as the percentage of children in the community who 
are Black increases from non-Black to mostly Black communities. Similarly, small changes in 
private school enrollment rates are observed for Hispanic and Asian students. As with White 
students, the overall patterns for the three non-White groups in 2000 are similar to 1990 (not 
shown, but available upon request). 

I extend the simple analyses above with regression models that include a variety of control 
variables. The first set of analyses is shown in Table 3 and includes results for 2000 for all four 
racial groups. The models for each racial group include the percent of children in a community 
who are Black (its quadratic term) and the percentage of children in a community who are of the 
same race as the student (also with its corresponding quadratic term). Other variables in Tables 
1 and 2 are not included in the models shown in Table 3 (and are added as control variables 
in subsequent analyses). Results indicate that in 2000 White students were more likely to be 
enrolled in private school as the proportion of children in their communities who were Black 
increased. This relationship is fairly linear, as indicated by the insignificant quadratic terms. Of 
importance, including the percentage of students who are White in the models for White students 


TABLE 3 
Random Intercept Logit Coefficients of Private School Enrollment Regressed on Racial Composition 
of Community, High School Students, 2000 


a 





Asian Black Hispanic White 
% of children in community who are —.004 (.006) —.019** (.005) .022** (.005) .018** (.004) 
Black 
% of children in community who are .0001 (.0001) .0001 (.0001) .0002** (.0001) | —.00002 (.0001) 
Black squared 
% of children in community who are —.022** (.008) —.017** (.005) .016** (.004) —.005 (.004) 


the same race 
% of children in community who are .001** (.0001) .0001* (.0001) .0001* (.0001) —.0001 (.00004) 
the same race squared 


Intercept ~2.509** (.079)  -2.388** (.139) -2.959** (.071) — -1.878** (.106) 
Random intercept —.494** (.131) —.506** (.092) —.283** (.076) —.475** (.037) 
N students 20,724 71,386 74,020 363,894 
Ncommunities 1,846 1,981 2,026 2,069 

Log likelihood —5,765 —11,928 —14,519 —106,293 

a 21 74 67 777 

p a Si) 155 186 159 

Op 781 .776 868 789 


ili crnnnnnn nn 
Note. For Black students, I have substituted the percentage of White children for the “percentage Black” variable. 

Standard errors are in parentheses. 

*p < .05. **p < .01 (two-tailed tests). 


184 S. SAPORITO 


(i.e., the “same-race” variables) does not render spurious the relationship between private school 
enrollment rates and the percent of children in the community who are Black. 

The coefficients in Table 3 also indicate that Hispanic children are more apt to enroll in 
private school as the proportion of children in the community who are Black increases. This 
relationship does not exist among Asian students. In contrast to other racial groups, Black 
students are less likely to be enrolled in private school as-the proportion of Black students in 
their communities increases. The magnitude of the coefficients in Table 3 must be interpreted 
with caution given that log odds do not lend themselves to intuitive interpretation. Although the 
coefficient for the “percentage Black” variable is statistically significant for Hispanic students, 
the substantive impact of this variable is much smaller for Hispanic students than White students. 
Similar conclusions hold for Black students with respect to the percentage of children in their 
communities who are Black. I discuss the magnitude of these effects more thoroughly in the 
context of models that control for community characteristics. 

The basic models shown in Table 3 demonstrate that private school enrollment is correlated 
with students’ racial backgrounds and the racial composition of the communities in which they 
live. These results are provocative but are based on simple models of private school enrollment 
that incorporate variables that measure community racial composition. These simple relationships 
are important in their own right: They might provide clues that some degree of racial segregation 
in neighborhood-based public schools is almost certainly attributable to private school choice 
patterns of White students. Still, these results do not demonstrate that the choices of (White) 
students are driven by race (even if the cumulative consequences of these choices may have 
the effect of exacerbating racial segregation in public schools). The correlation found between 
community racial composition and estimated private school enrollments among White students 
could be rendered spurious when available community-level characteristics are added to the 
models shown in Table 3. 

To address this issue, the models shown in Table 4 include a host of student, household, 
and community-level control variables. To be sure, available community-level control variables 
do not represent an exhaustive list of factors that families might consider when they choose 
between public and private schools. Still, the statistical and substantive conclusions change 
little when the battery of control variables is added to the analyses. Among White students the 
relationship between private school enrollment and the percentage of children who are Black 
remains statistically significant. By contrast, the percent of Black children is not significantly 
related to the probability that an Asian or a Black student is enrolled in private school. 

Because logistic regression coefficients do not lend themselves to intuitive interpretation, I 
estimate and graph the probability that members of each racial group are enrolled in private 
school as the percentage of children in their communities who are Black increases (while holding 
constant the percentage of children in the community who are of the same race as the student at 10 
and all other control variables at their means). Results are depicted in Figure 2. Both figures show 
that, among Asian, Black, and Hispanic students there is little association between the probability 
of private school enrollment and proportion of children in the community who are Black. (The 
intercepts for all racial groups are set to that for White children to allow for easier visualization 
of the marginal effects.) Indeed, the estimated proportion of Asian, Black, and Hispanic students 
who are enrolled in private school changes no more than .04 points (from the intercept to the 
inflection points) irrespective of the percentage of children in the community who are Black. 
Considered collectively, these patterns suggest that students who are non-White are not much 
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TABLE 4 
Random Intercept Logit Regressions of Private School Enrollment by Student Race, 2000 





Asian Black Hispanic White 

b SE b SE b SE b SE 
% Black .0003 .006 001 .005 .026** .005 10337 .003 
% Black squared 0001 .0001  -—00004 .0001  -.0002**  .0001  -.0001**  .00004 
% Same race —.020* .009 -.011 .006 —.008 .005 —.012** .004 
% Same race squared .0005** = .0001 0001 0001 .000 0001 .0001** — .00003 
Intercept —3.965** TAA ED 682** shi. 629) 2357844) S84 eA 227 2 321 
Random intercept —.785** 153° 5 FSUEhG7** Ue = 25% .092 —.946** .040 
N students 20,724 71,386 74,020 363,894 
N PUMS areas 1,846 1,981 2,026 2,069 
Log likelihood —5,264 —11,168 13,396 —101,863 
7 867 1,652 2,284 9,782 
p 22 086 128 .106 
On .675 558 .696 .623 





Note. Coefficients are based on models that include the following community-level control variables: Thiel’s 
entropy index segregation across census tracts within a community (H); dissimilarity between White and Black 
children across census tracts in a community; percentage of children who are not U.S. citizens; children per square 
mile; percentage of adults who are college educated; percentage unemployed; median household income; the 
Gini coefficient of income inequality. Student and family-level control variables include student gender and age, 
marital status of householder; family size; log of household income; whether householder owns home with no 
mortgage; whether householder rents dwelling; housing value; educational attainment of householders. A table 
with coefficients for all variables is available upon request. 

*p < .05. **p < .01 (two-tailed tests). 


more apt to enroll in private school as the percentage of children in their communities who are 
Black increases. 

In contrast to other racial groups, among White students there is a steady rise in the probability 
of private school enrollment as the proportion of children in the community who are Black 
increases. Specifically, the estimated percentage of White students enrolled in private school 
increases from 7 to 26% as the percentage of children in the community who are Black increases 
from 0 to 90. This 19-percentage-point increase demonstrates that even when available control 
variables held constant (including the proportion of children in a community who are of the same 
race) White students increasingly avoid sending their children to public schools in communities 
with higher shares of Black children. As with earlier results, private school enrollment patterns 
in 2000 are similar to those in 1990 (not shown, but figures are available upon request). 


Private School Enrollment: A Comparison of 1990 and 2000 


To explore potential changes in the correlation between race and private school enrollment over 
time, I explore whether the proportion of children in a community who are Black is as strongly 
associated with the probability of a White child enrolling in private school in 2000 as it was 
in 1990. To do this, I create variables that represent interactions between sample year and the 
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FIGURE 2 Estimated probability of private school enrollment by percentage of children in area who are 
Black holding all control variables at their means and the percentage of children who are of the same race at 
10, 2000. 


proportion of children who are Black. Specifically, I create a dummy variable for year (where 
the year 2000 = 1) and multiply this by a community’s racial composition. This interaction term 
essentially creates separate slopes for the proportion of children in a community who are Black 
for 1990 and 2000 and has the advantage of allowing me to incorporate racial composition slopes 
for both years in the same model. Incorporating 1990 and 2000 data in the same model allows 
me to assess whether any difference in the steepness of the community racial composition slopes 
across the 2 decades are statistically significant.'© 

Table 5 presents a series of models for White students. The first model presents a minimal 
specification that incorporates only the proportion of children in a community who are Black, 
the year, and the interaction between the year and community percentage Black. The results in 
Model 1 show clearly that the relationship between private school enrollment and the proportion 


10] eliminate the quadratic terms and “same race” variables from the 1990/2000 interaction models. None of the variables 
or their corresponding interaction terms undermines the substantive or statistical importance that Black community shares 
have on private school enrollment. Moreover, the quadratic terms and “same-race” variables do not reach conventional 
levels of statistical significance and are not enlightening in their own right. 
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TABLE 5 
Comparison of 1990 and 2000 Out-group Aversion Effects Among White High School 
Students 
Model 1 Model 2 Model 3 
b SE b SE b SE 
% Black 1027" 001 0205 001 02505 001 
Year 2000 one 041 —.657** in .609 468 
% Black x Year 2000 interaction -.001 .002 —.001 .002 .000 002 
Intercept EDO (030g 4.648745 — 2S5illes —Ss1G5A A SOS: 
Random intercept =135** 027 —.658** 029. -.701** 029 
Nstudents 694,399 694,399 694,399 
NPUMS areas 3,786 3,786 3,786 
Log likelihood —191,114 —182,874 —182,525 
x7 948 17,463 18,246 
p .210 136 Si 
On 935 720 704 


Note. Model 1 includes three variables (i.e., those shown in the table). Coefficients in Models 2 
and 3 are based on models that include the following community-level control variables: Thiel’s 
entropy index segregation across census tracts within a community (H); dissimilarity between 
White and Black children across census tracts in a community; percentage of children who are not 
USS. citizens; children per square mile; percentage of adults who are college educated; percentage 
unemployed; median household income; the Gini coefficient of income inequality. Student and 
family-level control variables include: student gender and age; marital status of householder; family 
size; log of household income; whether householder owns home with no mortgage, whether 
householder rents dwelling; housing value; educational attainment of householders. Model 3 
represents a fully saturated model with a full set of interaction terms that are not shown because 
of space limitations. A table with coefficients for all variables is available upon request. 

*p < 05. **p < .01 (two-tailed tests). 


of children who are Black is the same in 2000 as it was in 1990. This finding runs contrary to 
any expectation that race has declined as a social force influencing private school choice among 
White students. The fact that two nationally representative samples of children (taken 10 years 
apart) produce remarkably consistent results suggest that the findings are robust, leaving little 
doubt that race is correlated with private school enrollment rates. Moreover, the second and third 
models in Table 5, which incorporate a litany of control variables (with and without a full set of 
interaction terms), reduce the marginal effects in Model 1 modestly. 

To underscore the consistency of the results, Figure 3 illustrates the estimated association of 
school racial composition on private school enrollment rates based on the coefficients of models 
one and three of Table 5. There are four trends that represent pairs for 1990 and 2000. (The values 
of all control variables in Model 3 are set to their means for the subsample of White students.) In 
all respects, there is consistency in the estimates. The 1990 and 2000 pairings of Models 1 and 
3 are similar (although Model 3 reduces some of the racial effects). This suggests that economic 
factors in both years have some bearing on the bivariate correlations between community racial 
composition and private school enrollment. In these data, little has changed between 1990 and 
2000 and the decline in racial influences on school choice is statistically insignificant. 
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FIGURE 3. Estimated probability of White private school enrollment by percentage of children in area who are 
Black holding all control variables at their means and the percentage of children who are of the same race at 10, 
1990 and 2000. 


CONCLUSIONS 


Recent literature suggests there is greater racial segregation among students across local public 
schools than across their corresponding catchment areas. There are two theoretical explanations 
for this difference. Out-group avoidance suggests that historically privileged racial groups are 
more likely to avoid schools and other social settings as shares of disadvantaged racial groups 
increase. Same-race affinities suggest that the members of all racial groups will choose schools 
and other social settings with people who are like themselves racially. 

My empirical analyses suggest that private school enrollment rates likely contribute to greater 
racial segregation of students across school than of students across their corresponding school 
catchment areas and that this difference is likely the result of out-group avoidance patterns. 
Findings indicate clearly that there is a positive, strong, and consistent association between the 
probability that a White student is enrolled in private school and the percentage of students in 
a community who are Black. This relationship holds even when controls for the percentage of 
own-race students in the community are added. By contrast, there is, at most, a weak association 
between private school enrollment and community racial composition among Asian, Black, and 
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Hispanic students. These patterns are fairly consistent whether available individual-, household-, 
and community-level covariates are included in the analyses or whether 1990 or 2000 data are 
examined. 

Because I do not have information on the educational quality or safety of children in local 
schools, the religious background of parents or communities, or information that describes why 
some parents elect to live in integrated communities, I cannot conclude that the observed pattern 
of “outgroup” avoidance is due to parents’ reactions to racial composition per se. Nonetheless, the 
patterns clearly indicate that White parents opt out of public schools when they live in areas with 
increasing concentrations of Black students. The analyses here rely on private school enrollment 
patterns among children living in PUMS areas. These areas are larger than school catchment 
areas and thus serve as rough proxies of the area that families consider when deciding to remain 
in their local public school. Whether private school enrollment patterns observed across PUMS 
areas can be used to make generalizations regarding private school enrollment rates across school 
catchment areas is an open question. The higher propensity of White children to enroll in private 
school when they live in communities that have higher shares of Black children, however, is 
consistent with existing case studies that explore private or magnet school enrollment rates across 
school catchment areas (Bifulco et al., 2008; Lankford & Wyckoff, 2006; Saporito, 2003). 

What do my findings suggest about school choice in a postdesegregation world? Race, or factors 
strongly correlated with race, continues to have a significant influence on the choice of school 
for White parents. This fact suggests, that in the absence of policies designed to promote racial 
balance across schools, programs that expand the educational choices of families are unlikely to 
reduce racial segregation significantly. In so far as the recent ruling by the U.S. Supreme Court 
in Parents Involved in Community Schools v. Seattle School District (2007) limits the options 
districts have for regulating school racial composition, the prospects that school choice can serve 
the goal of promoting integration have been limited. 
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Reformers suggest that parental choice will improve equity by making it possible for parents to select 
better schools for their children. A key assumption behind this claim is that parents choose from a set 
of schools that range in quality. Data from this longitudinal interview study suggest this assumption 
may be false. In one Midwestern city, parents of different social class backgrounds did not consider 
schools of similar quality. The set of schools considered by parents, called the choice set, differed; 
though parents’ choice processes and reasoning were remarkably similar. These data suggest that in 
addition to the well-documented constraints of income, information, and transportation, the resources 
used to construct choice sets may further constrain the schools parents consider. These findings raise 
questions about the ability of current choice policies to deliver the equity outcomes reformers suggest. 


Across the country, poor children and children of color are “trapped” in inferior schools (Kozol, 
2005). Many are assigned to neighborhood schools that are poorly funded (Bifulco, 2005) and 
have the most inexperienced teachers (Lankford, Loeb, & Wyckoff, 2002). They often do not have 
transportation which would allow them to leave those neighborhood schools (e.g., Blank, Levine, 
& Steel, 1996; Nelson, Muir, & Drown, 2000). And they cannot afford housing in neighborhoods 
with better schools (e.g., Logan, 2002). These facts, widely agreed upon by scholars across the 
political spectrum, contrast sharply with the belief that all children, regardless of where they grow 
up or how much money their parents earn, should have access to public schools that support high 
academic achievement. 

One potential solution to this problem is parental choice. Increased parental choice will allow 
these families to escape and select better schools for their children (e.g., Friedman & Friedman, 
1980; Moe, 1995). This movement toward better schools will enhance equity by giving poor 
children and children of color access to a quality education. Finally, children will no longer be 
trapped. 

A key assumption behind this claim is the idea that parents choose among schools of varying 
quality. School choice theory rests on the idea that the set of schools from which parents select— 
their choice sets—must have at least some good schools. If parents’ choice sets contain only 
bad schools, school choice will allow them to escape, but their children will still be trapped in 
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inferior schools (Dillon, 2008). Thus, simply giving parents the opportunity to choose among 
many schools of similar low quality will not produce the equity gains choice promises. 

Rather than presume we know the set and the quality of schools from which parents choose, 
this study treats the choice set as an object of investigation. To better understand parents’ choice 
sets, I ask three related questions. First, what schools are in parents’ choice sets? Second, what 
processes do parents use to construct those choice sets? Finally, what factors shape choice set 
construction? 

I begin with a discussion of my use of rational choice theory, which draws on insights from 
the disciplines of sociology and psychology. I then describe the families who participated in the 
study and the methods used to conduct the investigation. Third, I describe parents’ choice sets and 
the processes they used to construct choice sets. Finally, I analyze three widely used resources 
for choice set construction and consider the implications for future research. 


THEORETICAL FRAMEWORK 


Although there is a great deal of research on the factors parents take account of in making their 
choices (Gill, Timpane, Ross, & Brewer, 2001), the processes parents use to choose schools 
have largely been uninvestigated by choice researchers (Hamilton & Guin, 2005). The level of 
specification in Bast and Walberg’s (2004) explanation of how parents choose is representative of 
the literature more generally. They explained, “Parents choose schools for their children based on 
costs and benefits (incentives), the availability of information, and the presence of opportunities 
(choices)” (p. 432). Like many other researchers, Bast and Walberg stop there. We know little 
about the fundamental aspects of the choice process (e.g., which schools are considered and how 
parents come to consider those schools). To better understand the choice process, I use insights 
from the literatures on bounded rationality, selection of postsecondary institutions, and social 
capital. These insights situate parents in their socio-historical realities thereby allowing for a 
contextualized explanation of their actions. 


Parents Have Bounded Rationality 


Herbert Simon’s Nobel Prize-winning work on decision making, bounded rationality, and satis- 
ficing is particularly relevant to parental choice. Simon’s work suggests that human beings cannot 
take account of all possibilities when they choose. They use heuristics and shortcuts; they have 
bounded rationality. Simon’s work also suggests that individuals do not necessarily choose the 
best option. Instead, they are “cognitive misers” and satisfice (Simon, 1986), “using experience 
to construct an expectation of how good a solution we might reasonably achieve and halting 
search as soon as a Solution is reached that meets the expectation” (Simon, 1990, p. 9). When 
parents choose schools, we might expect that they will not consider every school in their district 
nor will they necessarily choose the best school. We would expect, however, parents will select 
reasonable schools given their expectations of what is reasonable. 

Humans’ bounded rationality and the serious opportunity costs of actually considering every 
school to which one could reasonably transport their child require parents to consider some 
smaller set. The choice set is the set of schools a parent considers. It is defined subjectively by 
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the parent. Conceptually, the choice set is an analytic tool that describes and quantifies parents’ 
bounded rationality (e.g., Lau & Redlawsk, 2001; Lurie, 2004). The attributes of the choice 
set—its size, geography, prestige, and expense—provide insights into the heuristics and shortcuts 
parents use to simplify the decision-making process. 

It is important to note that the choice set is, by definition, subjective. Parents create their choice 
sets. A choice set may or may not reflect the schools close to a parent’s home. It may or may 
not reflect the variety of schools that actually exist at a given grade level in a particular city. By 
considering the content of the choice set and the processes used to create it, we are better able to 
understand how parents’ agency and market structures interact in parents’ decisions. 


The Construction of the Choice Set 


When a child reaches fifth grade and it is time to select a middle school, the parent does not 
suddenly invent a choice set. To varying degrees, the schools that parents are willing to consider 
evolve over time—something we must account for if we hope to understand parents’ actions. 
Hossler and Gallagher (1987) proposed a three-stage choice process that students use to choose 
an institution of higher education. I apply it here to K-12 parental choice to illuminate the 
development of parents’ choice sets. 

There are three stages in Hossler and Gallagher’s model: predisposition, search, and choice. 
The predisposition stage is when students decide if they want to continue their schooling beyond 
high school. If they so choose, students move on to the search stage in which they “gather 
information about institutions of higher education” (p. 209). During the search stage, some 
students also develop criteria for judging schools. The third and final stage, the choice stage, 
is when students actually decide on a college or university to attend. Hossler and Gallagher 
emphasized the interactions embedded in all stages, explaining that—throughout the stages— 
both structural and individual factors interact to create outcomes, which, in turn, influence the 
ultimate school selection. 

This three stage model is generative. It provides a straightforward set of categories to under- 
stand the process of constructing the choice set, and it emphasizes the structural and individual 
factors shaping parents’ actions. I use these three categories to study parents’ search behavior 
(predisposition and search), the resources they draw on during the search (search), and the reasons 
they give for their final decisions (choice). 


Resources for Constructing Choice Sets 


As parents set about constructing their choice sets and determining which school might be best for 
their child, they do so in a segregated, stratified social context. Parents do not have equal access to 
transportation, information, time for school visits, money for tuition, or English language skills. 
Resources, both material and immaterial, are not distributed evenly among parents of differing 
social class backgrounds. 

Studies of parental involvement find that social capital is a particularly important resource 
upon which parents draw to help their children succeed in school (Chin & Phillips, 2004; Holme, 
2002; Horvat, Weininger, & Lareau, 2003; Lareau, 1989, 2003). Social capital is “the material 
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and immaterial resources that individuals and families are able to access through their social ties” 
(Horvat et al., 2003, p. 323).! Social capital is important for many parent-school interactions, 
ranging from how parents negotiate school problems to the information parents seek out in school 
choice decisions. 

Parents’ social networks shape the strategies and resources available to address problems at 
school. For example, Horvat and her colleagues (2003) found that when faced with problems such 
as inappropriate teacher behavior or disagreeable academic placements, middle-class parents’ 
networks provided the information, authority, and expertise necessary to resolve disagreements 
in the parents’ favor. In contrast, working-class networks did not provide parents such resources 
and their negotiations with schools were less successful than middle-class parents. 

The information that flows across social networks is another type of social capital that has 
been linked to parents’ school selections (Holme, 2002; Schneider, Teske, Roch, & Marschall, 
1997). Holme’s interview study of mostly White, middle-class and upper middle-class parents 
found that parents’ schooling decisions relied almost solely on information from high status 
members of their social networks. Usually that information—which was often either partially or 
wholly inaccurate—concerned the reputations of schools instead of the curricular, test score, and 
other academic data about the school. Despite this, the information was highly valued. These 
studies, and others like them (e.g., Schneider et al., 1997), highlight both the importance of social 
networks and how uneven and inaccurate network information can be. 

Previous descriptions of K-12 choice have implicitly conceptualized parents as equals, all 
able to draw on similar resources to choose from the schools geographically available to them. 
The literatures on decision making, college choice, and parental involvement draw attention to 
the historical and social factors that may shape parents’ choice processes. Taken together, these 
literatures suggest the content and construction of choice sets will be shaped by the processes 
parents employ and their understanding and use of resources. 


METHODS 


To understand the composition and construction of parents’ choice sets, I conducted a longitudinal 
comparative case study of 48 parents’ thinking prior to their children attending sixth or ninth 
grade. I followed parents from January to November of 2003, interviewing them multiple times 
over the 9 months before, during and after they selected a new school. This allowed me to study 
the dynamic, situated nature of choice set development. It also allowed me to investigate the 
interactions between parents and the local choice market.” To my knowledge, this is the only U.S. 
study to follow parents longitudinally in this way. 

The study’s design purposely used diverse prior choices as a proxy for diverse parental thinking. 
Thus, it includes parents whose children had a wide range of schooling experiences. Within the 
sample, parents previously sent their children to both “failing” and “nonfailing” schools across 
six types of schools (neighborhood public, magnet public, charter, secular private, nonsecular 
private, and homeschool). By deliberately selecting a diverse group of parents, the 9-month study 
gave me access to a wide range of ways parents’ thought about the schools. The study’s design 


‘Social capital has been operationalized many ways; see Dika and Singh (2002) for a review. 
21 use the term “market” for simplicity; however, a more accurate term may be quasi-market (Henig, 1994). 
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does not allow for generalizations regarding particular groups. My goal is to describe the range 
of ways parents constructed choice sets and chose schools over time. 


The Context 


Like many cities and their surrounding suburbs in the Midwest, Weldon? is racially and eco- 
nomically isolated from its suburbs. Despite recent progress, the divide remains large. The city 
population is more than 88% people of color and has a median family income of approximately 
$33,000. In contrast, the adjacent suburbs have, on average, populations that are 18% people 
of color with a median family income of $62,000. Generally, the schools mirror this racial and 
economic segregation. Weldon schools are largely made up of students of color and students who 
qualify for free and reduced-price lunch, whereas many suburban schools are overwhelmingly 
White, with small proportions of students who qualify for the federal lunch program. 

There is a long history of choice in Weldon. However, until recently, choice was available 
only through residential movement or enrollment at tuition-based private schools. This changed 
dramatically a decade ago when the state passed its charter law. Since then, Weldon has been 
the site of much charter activity. As of 2001, parents living in Weldon and the adjacent ring of 
suburbs enjoyed choice options which include some 98 charter, 393 private, 48 magnet, and 341 
traditional public schools. Prior to the increase in charter schools and magnet schools, Weldon was 
a district with strong neighborhood school boundaries. Currently, the district is an option-demand 
system (Schneider, Teske, & Marschall, 2000); parents are assigned a neighborhood school based 
on their address, but they may request a transfer to another school if space is available at the 
receiving school. Although anecdotal evidence suggests this form of choice may be underutilized, 
there has been a recent outpouring of students to charters and neighboring districts—more than 
20,000 left between 1999 and 2005. 


Sampling 


The goal of the sampling procedure was to gather a group of parents that might think about 
choice in different ways. Because I could not know how parents would think about a future 
choice a priori, I constructed a sample of parents who had previously chosen schools with diverse 
characteristics. To gather schools that ranged in quality, all elementary and middle schools 5 miles 
on either side of the border between Weldon and the adjacent suburbs were classified as failing 
or nonfailing. A school was considered failing if, according to state accountability policy, it did 
not make adequate yearly progress (AYP) in 2000-01 or (in the case of secular and nonsecular 
schools) was unaccredited by any accrediting agency. Conversely, nonfailing schools made AYP 
or were accredited.* 


3 All names are pseudonyms selected by parents and students in the study. 

4Schools were accredited by a variety of accrediting bodies (e.g., Independent Schools Association of the Cen- 
tral States, North Central Association Commission on Accreditation and School Improvement). The vast majority of 
accreditation bodies require standards-based self-study and an accreditation visit by peers. 
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The distinction between failing and nonfailing is based on state accountability policies, but 
it is murky. There are a number of reasons it is complicated to judge school quality with these 
categories. First, we do not have a single metric against which all schools are judged. This 
means we must use different criteria for public and private schools. Second, if one assumes that 
standardized tests can measure school quality, private and religious schools need not participate 
in such tests nor must they make scores public, even if they do participate. Thus we do not have 
access to test scores for all schools. A final problem with the failing/nonfailing distinction is its 
validity. Because the criteria upon which states judge schools are themselves political objects, 
they may or may not be valid measures of school quality. Across the country there is a great deal 
of discussion about how states determine when a school is failing and whether that designation 
actually relates to problems significant enough to deserve sanctions (e.g., Winerip, 2005). Despite 
all this, No Child Left Behind imposes severe consequences on failing public schools, such as 
allowing students to transfer to other schools. Further, policymakers presume this label has 
meaning to parents regardless of its validity. So although this distinction is analytically weak, it 
affects the schools available to parents. 

L use this distinction as a crude way of tracking the schools parents consider. It is not clear 
how failing and nonfailing schools differ; however, I apply the definition to all schools. Thus, 
as a marker, the distinction allows me to track schools across parents’ choice processes. I track 
other markers as well (i.e., selectivity, tuition requirements). 

After classifying schools in the border region of Weldon, a purposeful sample of one failing 
and one nonfailing school was selected for each of five school types (neighborhood pubic, magnet 
public, charter public, secular private, and nonsecular private). School principals were asked to 
supply a list of fifth (or eighth) graders’ addresses. By using 2000 census data, each address 
was assigned a median family income based on its block group. Addresses were then stratified 
into low, middle, and high income categories.® Potential participants were randomly selected 
from each of the three income groups and contacted by telephone. This resulted in a sample of 
48 parents stratified by their current status at a failing or nonfailing school and median family 
income. Parents with similar characteristics replaced those parents who declined participation. 
The positive response rate was 60%, the nonresponse rate 32%, and the negative response rate 
8%. The sample of 45 mothers and 3 fathers was comprised of 67% African Americans, 27% 
Whites, 4% Hispanics, and 2% people of other races. The racial profile of the sample was similar 
to the racial profile of the city; however, it overrepresents white parents (by approximately 
13%). 

The purpose of this sampling strategy was to gain access to the range of ways Weldon 
parents might construct choice sets. As such, the study offers new conceptualizations and 
descriptions of the choice process but it cannot support claims about which schools cer- 
tain groups (i.e., parents in neighborhood schools, middle-class parents, etc.) will consider or 
choose. 


Exploratory analyses suggest that the patterns I find are robust even when I recategorize schools based on how many 
years they did not achieve AYP or how long they were unaccredited. 

6Less then $39,000 was considered lower income, $39,000 to $88,000 was considered to be middle income, and more 
than $88,000 was considered to be upper income. These cut points were determined by dividing the income distribution 
in Weldon and the inner ring of suburbs into thirds so that one third of people fell into each category. 
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Data Collection and Analysis 


Data collection and analysis were iterative (Miles & Huberman, 1994). I used three interviews, 
which spanned the 9 months preceding and immediately following the selection of a new school, to 
capture and track parents’ choice processes and choice sets. After each wave of data was collected, 
I transcribed all the interviews and made data representations that were member-checked with 
parents. 

The first interview occurred in February (face-to-face), the second in June (face-to-face), and 
the third in October (phone). The interviews averaged 60, 49, and 34 min, respectively. All told, 
112 hr of interviews were conducted with the 48 parents. All interviews were audio-recorded. 
Parents were compensated for their participation with a $30 gift certificate to a store of their 
choice. 

At the end of data collection, transcripts were read into the qualitative software, N6, and coded 
(see Bell, 2004, for coding details). Coded data were then tabulated, entered into a Microsoft Excel 
spreadsheet, checked for accuracy, and imported into the statistical software package SAS. Once 
in SAS, descriptive statistics were generated around issues of parental reasoning (e.g., the number 
of schools considered and applied to, the type of choice process, etc.). As discrepancies emerged 
and additional categories were needed, the process was repeated. This process of analyzing data 
allowed hypotheses to be tested, accepted or rejected, and revised as necessary. Simultaneously, 
I wrote analytic memos about patterns and themes that arose while reading and coding the data. 
These memos were used as the catalyst for discussions with colleagues; those conversations, and 
the SAS and N6 data analyses, sent me back to the data for additional rounds of analysis. 

I operationalized “choice sets” by noting each school a parent mentioned. The schools held 
different status in parents’ minds and across the study’s 9 months. Some schools were mentioned 
and rejected quickly. Others remained viable over the course of the study. Still others were 
mentioned as nonoptions. I recorded each school considered and the parent’s thoughts about that 
school in a central database. The longitudinal nature of the study allowed me to document the 
differential status particular schools held within parents’ choice sets, as well as the evolution of 
choice sets over time. 

As analyses revealed the importance of social capital in the construction of choice sets, parents 
were assigned a social class status—poor, working class, or middle class—based on self-reports of 
income, educational attainment, and autonomy in the workplace. The details of this categorization 
are described in Bell (2008b). Because this was a relatively small study, these three broad groups 
captured the range of social class experiences among the parents. This is consistent with other 
studies of this size and scope (e.g., Lareau, 1989, 2003). I define social class functionally—as a 
group that possesses a similar income level, autonomy in the workplace, and educational level. I 
do this in an attempt to take account of the probabilistic nature of the construct (Smith, 2007). By 
using all three factors in my designation, I took advantage of the incremental validity associated 
with more complex and robust measures of social class (e.g., Krieger, Williams, & Moss, 1997). 


CONSTRUCTING THE CHOICE SET 


Parents of different social class backgrounds used similarly diverse strategies for constructing 
their choice sets. Parents used one of three search processes: They elected not to search for a 
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school, they conducted a closed search, or they looked extensively using an open search strategy. I 
describe each next. Once parents had constructed a choice set, their reasons for selecting a school 
were similar to those documented in the literature, with the overwhelming proportion of parents 
citing academic and holistic reasons. Though there were small differences, parents across social 
classes reported similar reasons for choosing schools. 


The Predetermination Phase: Parents Who Do Not Search 


As mentioned previously, Weldon parents are in an option-demand choice environment. Of the 
48 parents in the study, 40 previously opted out of their neighborhood schools. But substantially 
fewer opted out of customary enrollment patterns, that is, the patterns made when children attend 
an expected sequence of schools (elementary, middle, and high). These patterns are comprised 
of what Fiske and Ladd (2000) called feeder schools. The customary enrollment pattern, often 
unofficial and unwritten, provides a ready-made next school(s) for most parents. These patterns 
exist in both the private and public sector. In Weldon, for example, if you go to St. Mary’s for 
middle school, it is expected that you will either go to St. Patrick’s or Atonement High School. 
The moves from St. Mary’s to either St. Patrick’s or Atonement define two customary enrollment 
patterns for St. Mary’s families. Thus, given parents’ previous choices, in the predetermination 
phase they have to answer a question: Do I need to conduct a search? The answer depends on the 
parent’s preferences regarding both the assigned school and the customary school(s). 

Among the 48 Weldon parents, 33 conducted a search and 15 did not. The 15 parents who did 
not conduct a search were not lazy or disinterested, for they talked extensively and exhaustively 
about their decision making. They chose not to conduct searches for a range of reasons. The 
majority of parents (9 of 15) reported that no other schools offered what they wanted. These 
parents knew the local schooling market from prior searches conducted for older siblings, felt 
confident that the school in the customary enrollment pattern was what they wanted, and were 
sure that other schools would not be able to offer what the customary school did. A few parents 
said that the customary school was a good school with a good reputation, so they were willing 
to try it. The remaining parents reported that they were comfortable with the customary school 
and would reassess their decision after the next school year. Although one third of parents did 
not consider more than one school, effectively ending the construction of the choice set at the 
predetermination stage, approximately two thirds of parents did search for a school. Their actions 
help us understand how larger choice sets get constructed. 


The Search Phase: Open and Closed Searches 


There were two search procedures by which parents came to select a school for their child. The 
first, the open search, began with a larger set of schools and over time—in interaction with the 
schooling market—narrowed to some smaller set of schools from which the parent ultimately 
selected. The second search procedure, a closed search, was a process in which the parent had 
two, sometimes three, schools in mind. The parent did not seek out other schools to consider 
and, instead, spent time gathering information about those two to three schools and making the 
necessary preparations for the child to attend the schools. Preparations included actions such as 
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taking entrance exams, organizing finances, and arranging transportation. The major differences 
between an open and a closed search were the number of schools considered and the desire to 
find additional alternatives to consider. Of the 33 Weldon parents who conducted a search, 11 
performed a closed search and 22 performed an open one. On average, closed searches contained 
3.4 schools, open searches contained 7.5 schools. 


The Choice Phase: Reasons for Choosing 


Parents gave 102 different reasons for choosing the school their child ultimately attended. These 
reasons were coded into six categories: holistic, academic, social, logistic, administrative, and 
“other.” Holistic reasons focused on the child’s overall well-being, including reasons such as “They 
are thriving where they are” and the child “isn’t ready for that kind of school.” Academic reasons 
were those that focused on concerns around classroom teaching and learning. The curriculum 
being “too basic,” “good teachers,” and “many learning resources” were all coded as academic 
reasons. Parents mentioned many social reasons as well, for example, “Friends are going there” 
or the child “knows people there” or the students are “too rough” at that school. Social reasons 
focused on relational concerns. Logistical reasons were those that pertained to the location, 
transportation, and cost issues. For example, reasons such as “moving to the area,” “too expensive,” 
and “close enough to [the sibling’s] school” were all coded as logistical reasons. In contrast, 
reasons such as “couldn’t meet some of the school’s paperwork requirements” or “missed the 
application deadline” were coded as administrative reasons, as they related to how the school was 
organized and run. Finally, there were a few reasons that did not fit well into any of the other 
categories. Reasons such as “that school has good lunch” were coded into the other category.’ 

In parents’ final school selection, a majority of Weldon parents sited holistic (69%) and aca- 
demic (58%) reasons for choosing a school. This is consistent with other research that describes 
parents as concerned with their children’s overall development, especially as that development 
relates to the academic characteristics of schools (e.g., Fiske & Ladd, 2000; Gill et al., 2001; 
Schneider et al., 2000). Considerably fewer parents cited social (33%), logistic (27%), adminis- 
trative (25%), and other (1%) reasons. 

Given the importance of race in the United States and school choice research, it is worth noting 
how it played out in parents’ choice sets and final reasoning. With a handful of exceptions, city 
parents were people of color and their choice sets contained only schools in which almost all 
students were students of color. Suburban parents were White and their choice sets contained 
schools in which almost all students were White. Many parents said they wanted more diverse 
schools, but the lack of variation in the racial composition of choice set schools and the cities in 
which parents reside suggest race issues played out in more subtle ways. Preliminary analyses 
suggested that a city parent might, for example, choose amongst choice set schools that were 97, 
98, and 99% African American. Thus, the differences in choice set schools were so small there 


7Parents discussed safety in different ways. On surveys “safety” is a comprehensive category that obscures the subtlety 
of the ways in which parents use the term. For example, some parents talked about unsafe schools as places that threaten 
their child’s ability to develop, socially and intellectually. Other parents discussed unsafe schools as socially chaotic 
places in which their child might get “mixed-up” in the wrong crowd. Depending on the context in which the parent 
discussed their concerns, “safety” was coded in the social, holistic or administrative category. 
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TABLE 1 
Proportion of Parents in Predetermination, Search, and Choice Phases 
Phase of Search Process All Poor and Working Class Middle Class 


Predetermination 


No search 31 24 39 
Search 

Closed 3 24 oy 

Open A6 52 ooo 
Choice 

Holistic 69 64 74 

Academic 58 68 A8 

Social 35 36 30 

Logistic Dh, oe ee 

Administrative aS oo ally 
n 48 25 23 


evga ois Sea eS 


Note. Exact chi-square tests assessed the relationship between features of the search 
phase and social class group. 
op —.05; 


was little to study once parents constructed a choice set. That said, race played an important role 
in the preferences that operated prior to choice set construction (see Bell, 2007, 2008b). 


Similar Choice Processes and Reasoning 


Although there was remarkable agreement on the criteria used to choose a school, parents 
constructed their choice sets in variable ways. These differences in choice set construction were 
not associated with social class background. Parents constructed choice sets in similar ways and 
reported similar reasons for selecting schools for their children. Between one fourth and one 
third of parents did not conduct a search and, thus, did not move past the predetermination stage. 
The majority of parents from both social-class groups conducted some type of search (61% for 
middle class and 76% for poor and working class).® Those searches culminated in a choice that 
was most frequently attributed to holistic or academic concerns (see Table 1). 

Despite these similarities, there were differences between groups of parents. For instance, 
a greater proportion of poor and working-class parents conducted open searches than middle- 
class parents. Poor and working-class parents considered a larger number of schools than more 
advantaged parents. Poor and working-class parents also cited academic reasons for choosing 
their final school more frequently than did middle-class parents. These differences were not, 
however, statistically significant. 


8This pattern is consistent with one report of non-White parents who receive free lunch subsidies (i.e., poor parents) 
articulating preferences for more schools than non-White wealthier parents (Hastings, Kane, & Staiger, 2008). 
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TABLE 2 
Characteristics of Mean Parent's Choice Set and Final School Selection, by Social Class 





Choice Set Final School Selection 
Poor and Poor and 

Characteristic Working Class Middle Class n Working Class Middle Class 
Nonfailing* .38 (.37) 25 .65* (.38) 19 36° 3% 
Selective* 31 G32) 25 AT.) 23 24> 74% 
Tuition based* 14 (.29) Ds) 50** (40) 23 24 43 
At least 2 nonfailing schools” .16 pe) Bane 19 
No. of schools* 4.3 (3.0) 25 B39) eee 


“Two-tailed independent ¢ tests assessed the difference between mean proportions. ?Exact chi-square tests 
assessed the relationship between the choice set characteristic and social-class group. Standard deviations are in 
parentheses. 

iOS at pee () 1, 


PARENTS’ CHOICE SETS 


Similar means led to different ends for the 48 Weldon parents. Despite the fact parents across 
social classes used similar processes and reasoning, there were important differences in the types 
of schools parents considered (Table 2). 

Middle-class parents’ choice sets contained, on average, a greater percentage of nonfailing, 
selective,’ and tuition-based schools than did poor and working-class parents’ choice sets. In 
addition, just 16% of poor and working-class parents had at least two nonfailing schools in their 
choice sets, whereas 58% of middle-class parents had at least two. The differences between 
parents’ choice sets were consistent with differences between the schools parents ultimately 
selected. 

In August, 53% of middle-class parents chose a nonfailing school as compared with 36% of 
poor and working-class parents. Middle-class parents chose selective and tuition-based schools 
at higher rates than poor and working-class parents. Although a greater proportion of middle- 
class parents chose nonfailing, selective, and tuition-based schools than poor and working-class 
parents, I do not know if they chose schools of better academic quality. Comparative data (e.g., 
standardized test scores, teacher qualifications, etc.) were publicly unavailable for almost half of 
the schools parents selected. Thus, it is possible that middle-class parents chose better schools, 
but these data only support the claim they chose different schools. 

To summarize, both groups of parents used similarly diverse strategies to choose. Poor and 
working-class parents sought out schools and reasoned in ways similar to more advantaged 
parents. They considered more schools and cited academic reasons more often than middle-class 
parents. Yet poor and working-class parents’ choice sets and final school selections included a 
higher proportion of failing, nonselective, free schools than those of middle-class parents. In 
short, similar strategies resulted in different outcomes. Social class differences in parents’ choice 


9Selective schools are schools that have competitive entrance requirements, require an entrance exam or a financial 
commitment beyond paying taxes. Magnet, secular, nonsecular, and homeschools are in this category. 
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sets may not be particularly surprising, however, they have not yet been documented in the 
literature. 

Previous work has documented how less than optimal parental decision making and market 
shortcomings (e.g., inadequate school supply, transportation) can account for parents’ decisions 
to select seemingly undesirable schools, such as some of the ones selected in Weldon (Arsen, 
Plank, & Sykes, 1999; Glatter, Woods, & Bagley, 1997; Schneider et al., 2000). Although these 
constraints certainly played a role, the nature of the resources parents used to construct their 
choice sets acted as an additional constraint on parents’ considerations. 


CLASS-BASED DIFFERENCES IN RESOURCES FOR CONSTRUCTING 
CHOICE SETS 


When constructing their choice sets, poor and working-class parents used the same resources as 
middle-class parents but encountered a smaller proportion of nonfailing, selective, and tuition- 
based schools. Three social and historical resources were crucial for choice set construction across 
social classes. Social networks, customary enrollment patterns, and students’ academic histories 
shaped all parents’ choice sets, but did so differentially. 


Social Networks 


Previous analyses suggest that social networks can provide parents with many resources for 
choosing schools (e.g., Schneider et al., 1997). This was true for Weldon parents. They used their 
networks to gather information about particular schools, learn about new schools, and gather 
multiple perspectives on a single school. With few exceptions, parents placed a great deal of 
importance on the value of information gathered through these networks. 

Of the 48 parents, all but 3 used their social networks to learn about schools. This high level 
of use was similar across social classes. Networks tended to be comprised of family, friends from 
church, neighbors, and parents of their children’s friends.’ Network members nominated more 
than three fourths of all the schools in choice sets. These networks did not, however, provide 
access to similar schools. 

Overall, middle-class parents’ social networks put them in contact with a higher proportion of 
nonfailing, selective, and tuition-based schools than did poor and working-class parents’ networks 
(see Table 3). Middle-class parents like Mrs. Brown discovered potential schools through her son’s 
friends’ parents. Parents at Sebastian’s yearlong Saturday technology classes were particularly 
trusted and knowledgeable. She explained, “[The other parents] know every school in the city. 
When Sebastian was in session, we all would talk. That’s where I get a lot of my information.” 
Mrs. Brown’s choice set contained four schools, three nonfailing, three selective, and one tuition 
based. Working-class and poor parents also talked about schooling and schools with people in 
their networks; however, the characteristics of those schools differed substantively. One of the 
poor mothers, Mrs. Gooden, asked friends, family members, and parents of her daughter’s friends 
what schools they were considering and any information they had about those schools. This 


10No data were collected on social network members’ race, income, ethnicity, or child’s academic performance. 
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TABLE 3 
Characteristics of Mean Parent’s Social Network, Customary Enrollment Pattern, and Child’s 
Academic History, by Social Class 


Contextual Factors Poor and Working Class n Middle Class n 


Social networks 


Schools nonfailing* 48 (.32) 23 .64 (.43) 18 
Schools selective* 31 (.36) 23 .90** (.19) 22 
Schools tuition based* 14 (.31) 23 .64** (.42) DD 
Customary enrollment pattern 
Schools nonfailing® 44 (.47) 24 57 (.49) 15 
Schools selective* .23 (.36) 24 Ste CSI) 15 
Schools tuition based* .10 (.29) 24 .50** (.46) 15 
Child’s academic history 
A or “mostly” A student” A0 25 BS Oma 23 
B student” 48 25 17** 23 
Successful academic year? 56 Da .87* 23 
Students retained* 12 25 .00 23 





4Two-tailed independent t tests assessed the difference between mean proportions. "Exact chi-square 
tests assessed the relationship between the contextual factor and social-class group. Standard deviations 
are in parentheses. 

SpaseOdat tp <0 


resulted in a choice set of four schools, zero nonfailing, two selective, and zero tuition-based. 
The differential access to high quality schools these examples illustrate is noteworthy given the 
large proportion of schools nominated by social networks. 


Customary Enrollment Patterns 


As mentioned earlier, customary enrollment patterns exist in all school sectors (except home- 
schools). In charter, secular, and nonsecular schools, there tended to be more than one school in 
the customary pattern. In neighborhood schools, there was usually a single school in the pattern. 
Overall, schools in the customary enrollment pattern accounted for 49% of the schools in parents’ 
choice sets. Though Weldon parents were not bound to select the next school in the customary 
pattern, many did. The tidal pull was strong.'! 

In many cases preferences around these schools were important in the decision to search 
at all. One middle-class mother, Mrs. Erhardt, explained her thinking about where to send her 
eighth-grade daughter. “All of her friends will go to St. Christopher. But I have some concerns. So 
I’m looking into other options. I’m not sure if we’ll find something we like better.” The presumed 
school was St. Christopher. Had Mrs. Erhardt been comfortable with the school, her choice set 
would have been a single school. She would not have searched at all. Similarly, Mrs. Bordon 


!1 These customary enrollment patterns do not necessarily keep parents close to home. On average, middle-class parents 
traveled 4.6 miles and poor and working-class parents 4.9 miles to their selected schools. Geography played an important 
role in choice (Bell, 2007, 2008a), however, because of space constraints, I do not explore that role here. 
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considered a noncustomary school only when her eighth-grade son said he did not want to attend 
the school in the customary pattern. After thinking about her own experience being forced to go 
to a high school she did not want to attend and the risks involved in forcing her son to do the 
same, Mrs. Bordon allowed her son to opt out of the customary pattern. Had Mrs. Bordon’s son 
been comfortable with the customary school, she would have happily sent him there. 

Parents across social classes selected schools in the customary school enrollment pattern 
at similar rates; 52% of middle-class and 56% of poor and working-class parents. Customary 
enrollment patterns did not, however, provide contact with similar schools (Table 3). Paralleling 
the contact provided by social networks, poor and working-class parents’ customary enrollment 
patterns provided little access to nonfailing, selective, and tuition-based schools. 

The evidence from social networks and customary enrollment patterns suggests that parents 
across social classes drew on these resources at similar rates. Despite this, middle-class parents 
had greater contact with nonfailing, selective, and tuition-based schools. Contact with a school 
through social networks or the customary enrollment pattern does not necessarily ensure that a 
parent will choose one of those schools. Thus, we consider a third factor that shaped parents 
choice sets: children’s academic histories. 


Children’s Academic Histories 


All children’s academic histories shaped the types of schools parents perceived to be “real” 
options. There were, however, class-based differences in those academic histories. 

Mrs. Carol’s thinking shows how parents connected children’s academic histories to the schools 
they considered. Her thinking is representative. She explained how her eighth-grade son Denzel’s 
prior achievement shaped her thinking about where he should go to high school: 


[If] I’m sending him to public school for free and he’s going there and he’s not putting forth no effort, 
then I would really be fighting a losing battle to spend all my money and send him over there [pointing 
to a private school down the street] . . . "cause you’ve got to be motivated to learn, you know, and that 
starts from within. And I’m working on getting him motivated. And that’s hard. 


Mrs. Carol made a connection between her child’s historical achievement and future school. 
She concluded that it would not make sense to send Denzel to private school; he was unlikely to 
dramatically change his academic achievement. In her mind, sending him to private school would 
transform a challenging problem into a challenging and expensive problem. Mrs. Carol did not 
consider private or religious schools for Denzel, nor did she consider magnet schools because 
“with his grades, he wouldn’t get in.” Mrs. Carol’s thinking exemplifies the type of connections 
that parents made between achievement history and potential schools. 

Selective and tuition-based schools were often perceived to be “better,” “more challenging” 
schools. Children with stronger academic histories were seen as more likely to do well in a 
challenging school. Their parents tended to consider more selective or tuition-based schools. 
Parents whose children had less positive academic histories generally explained that they did not 
want to set their children up for failure. Parents who backed away from “challenging” schools 
did not think they were choosing from a set of inferior schools. Instead, they thought the schools 
they considered would meet their child’s needs if the child put forth the appropriate effort. It is 
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worth noting that there were a handful of parents who deliberately chose a challenging school in 
an attempt to gain access to better teachers and more resources for their academically struggling 
child. But this was not the norm. 

Consistent with national trends, poor and working-class Weldon children were less academ- 
ically successful than their middle-class peers (Table 3).!* Although parents of all social-class 
backgrounds were generally positive about their children, middle-class parents reported higher 
achievement levels than did poor and working-class parents. Middle-class parents had 5.2 greater 
odds of reporting their child was having a successful academic year as compared with poor and 
working-class parents. Middle-class parents also had 7.1 greater odds of reporting their child 
earned mostly As or As and Bs, as compared with poor and working-class parents.!? 


DISCUSSION 


This article set out to open the black box of the choice process by looking at parents’ choice 
sets, the construction of those choice sets and variation across parents. It has shown that parents 
of varying social class backgrounds had different choice sets that were not explained by the 
processes or reasoning they used to construct their choice sets. Instead, the resources parents 
used to construct their choice sets constrained the schools they were willing to consider along 
existing boundaries of social inequality. Although it remains important to scrutinize the number 
of high-quality schools available to parents, these findings suggest we must also scrutinize the 
constraining influence of the choice set so that all parents consider the options that are available. 

These findings support the importance of both heuristics and bounded rationality in under- 
standing parents’ choice processes. No parent considered all the grade appropriate schools in 
Weldon. Every parent’s search was bounded in some way. The application of Hossler and Gal- 
lagher’s model of the choice process suggests that some parents did not search at all—stopping at 
the predisposition phase—some searched a little, and others searched exhaustively. In the search 
and choice stages, parents’ processes and reasoning were similarly diverse across social classes. 
These findings raise an important question: How do similar choice processes and reasoning lead 
to different choice set composition for parents in different social classes? 

The answer has to do with the socio-historical nature of choice. Specifically, the resources 
most used by parents to construct choice sets—social networks, customary enrollment patterns, 
and understanding of student achievement—were inextricably linked to the current distribution 
of educational opportunities. This distribution is uneven and disadvantages poor children and 
children of color (e.g., Oakes, 1990). In the case of social networks, middle-class parents’ 
networks included education professionals, doctors, lawyers, and so on, who are likely to have 
the type of social capital (e.g., information and prestige) that facilitates school choice. With 
respect to enrollment patterns, poor and working-class children already attend schools that have 


12Their success was measured both by the grades parents’ reported and by parents’ overall assessment of the child’s 
academic success in the current school year. 

13The odds of an outcome is the number of “A” outcomes divided by the number of “B” outcomes. The odds ratio is 
the odds for the first group divided by the odds for the second group. An odds ratio of 1 indicates the outcome is equally 
likely in both groups. An odds ratio greater than 1 indicates the outcome is more likely in the first group. And an odds 
ratio less than 1 indicates that the outcome is less likely in the first group. Odds ratios are used as an effect size indicator. 
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lower teaching quality and offer increasingly impoverished curricular options (Kozol, 2005). 
Customary enrollment patterns will include more of these schools for poor and working-class 
children than middle-class children. Finally, children of color and White and Asian children do 
not achieve at the same levels (National Center for Educational Statistics, 2001). Thus, parents’ 
assessment of their children’s academic history operates in a system that advantages middle-class 
students and their families (Delpit, 1995; Lareau, 1989, 2003). 

It seems relatively innocuous to note that parents’ choices are bounded. But when you consider 
how existing social inequalities shaped those bounds through the almost invisible influence of 
resources, the significance of that bounding becomes clearer. The structural inequalities in social 
networks, enrollment patterns, and academic histories are compelling and under-documented 
factors in parents’ decision making. Even if choice causes schools to improve across the board, 
we have no reason to believe these resources will operate more equitably or become less relevant 
to choice set construction. 

The extent of this bounding is also important given the purposeful sample upon which this 
study is based. The sample is biased toward parents who opted out of their traditional public 
schools, and as such, it likely overestimates the average parent’s agency in school choice. If 
factors associated with social class play an important role in these parents’ choice sets, it is 
possible that the choice sets of parents who have not historically opted out of traditional schools 
may differ even more dramatically along social class lines. This possibility raises questions about 
how parents’ choices are going to reform our schooling system. Further, it suggests that policy 
makers should consider ways to intervene in the structural inequality inherent in the resources 
parents draw on to construct choice sets. For example, directing accurate schooling information, 
which addresses parents’ concerns about prior achievement, academic excellence, and holistic 
views of children, into church, neighborhood, and school networks may decrease the inequalities 
documented in these parents’ choice sets. 

This study also generates new questions. These questions will require studies that make use of 
large, representative samples to explore the demand side of the market. Such studies would allow 
us to answer questions like: Do choice sets and their construction differ by sector (i.e., private, 
public, etc.)? How do local geographies and choice policies shape the demand for particular forms 
of school choice? What do choice sets look like in representative samples? As scholars move 
forward, it will be instructive to consider recent scholarship on the role of parental involvement in 
academic achievement. Increasingly complex applications of social and cultural capital theories 
have facilitated new understandings in this area (e.g., Horvat et al., 2003; Lareau, 2003) and the 
application of such theories to parental choice may be similarly generative. 


CONCLUSION 


For choice to enhance equity, parents must select from schools of varying quality. This study 
found that poor and working-class parents did not choose from schools that ranged in quality. The 
schools they selected from were relatively uniform: failing, nonselective, and free. Middle-class 
parents’ choice sets were relatively uniform as well: nonfailing and selective. These data suggest 
that one assumption upon which the logic of choice rests may be flawed. Further, the study 
nominates an additional constraint on parental decision making—inequality in the resources 
parents use to construct choice sets. 
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Schooling markets are often perceived to be open, unbiased, and fair. It is suggested that if 
we simply give parents choice, they will select the best school from the set that exists. But this 
straightforward logic does not account for the social and historical factors that shape parents’ 
decision making. Although the supply of quality schools matters, if choice is to deliver significant 
equity gains, our policies must take better account of the almost invisible social and historical 
inequities that constrain the schools parents are willing to consider. 
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This article uses qualitative case study methodology to examine why the racial composition of magnet 
schools in Nashville, Tennessee, has shifted to predominantly African American in the aftermath of 
unitary status. The article compares the policy contexts and parents’ reasons for choosing magnet 
schools at two points in time—under court order and under unitary status. Social networks, choice 
sets, and “closer to home” student assignment policies are identified as key influences on the changing 
demographics of magnet schools in a postunitary status environment. 


Magnet schools, sometimes referred to as “alternative schools” or “schools of choice,” are 
public schools that provide incentives to parents and students through specialized curricular 
themes or instructional methods. The term “magnet” gained popularity in the 1970s when policy- 
makers were designing desegregation plans in an effort to make them more attractive to parents, 
educators, and students. Magnet schools provide school districts with an alternative to manda- 
tory reassignment and forced busing by providing a choice for parents among several school 
options, each offering a different set of distinctive course offerings or instructional formats. In 
magnet schools, enrollments are often managed to ensure a racially balanced student population, 
as directed by the supervising district judge. 

This article documents the shift to racial imbalance (half of all Nashville magnet schools 
are 75% African American) after unitary status and the adoption of race neutral admissions 
policies. The central question is, Why did racial composition in thematic magnet schools tip to 
predominantly African American in the aftermath of unitary status in Nashville? I explore the 
influence of magnet school geographical location and neighborhood demographics on parents’ 
decisions to choose (or not) magnet schools. In short, the evidence suggests that middle-class 
African American families from across the district are choosing predominantly African American 
magnet schools whereas White families perceive the magnet school options as unattractive under 
conditions in which, as one magnet school principal noted, “their child will be the only white kid 
in the class.” 
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UNITARY STATUS, RACE POLICIES, AND MAGNET SCHOOLS 


The removal of mandatory desegregation strictures is known as a grant of “unitary status.” 
Once a district is declared unitary, priorities and policies focus on expanding choice options, 
neighborhood schools, and quality improvement, replacing the previous emphases on judicial 
oversight and racial balancing. 

In a series of major federal court rulings during the 1990s, school district efforts to maintain 
admission policies that are designed to promote and ensure racial diversity in magnet schools were 
repudiated.!' These rulings forbid districts declared unitary from using separate race-based lotter- 
ies in magnet school admissions—formerly a well-established mechanism designed to produce 
race-balanced enrollments in these schools (Boger & Bower, 2001). During 2003-2005, district 
and appeals court rulings across the nation (e.g., Lynn, MA; Louisville, KY; and Seattle, WA) 
pointed to a wide range of race-conscious options available to districts that pursued voluntary 
magnet school and student assignment policies linked to school diversity goals. 

The legal ambiguity regarding the Constitutional limits on the use of race in school desegre- 
gation polices was clarified on June 28, 2007, in one of the most consequential U.S. Supreme 
Court rulings on race and education since Brown in 1954. In Parents Involved in Community 
Schools (PICS) v. Seattle School District, the Court limited the use of race in student assignment 
and school choice plans. The decision directly impacts districts currently operating under unitary 
status, like Nashville, but does not affect districts functioning under existing court-ordered de- 
segregation plans. The case has critical significance, too, for districts that are planning to pursue 
unitary status because these districts would be subject to PICS restrictions on the use of race in 
voluntary plans designed to create racial diversity under post unitary status. 

Justice Kennedy’s concurring and controlling opinion underscored that a district may con- 
sider it a compelling interest to achieve a diverse student population and offered the following 
mechanisms designed to achieve the aims of racial diversity: (a) devising student attendance 
zones to encompass racially defined/segregated neighborhoods, (b) building new schools in 
racially mixed neighborhoods or in areas that straddle racially identifiable neighborhoods, and 
(c) developing special or unique programs. The problems (or challenges) associated with these 
approaches are well known and well understood. The first remedy is often associated with 
cross-town busing plans that assign students from racially segregated city neighborhoods to a 
school many miles away; this plan is antithetical to most parents’ priorities of close proximity 
between school and home and one of the central reasons that districts seek relief from costly, 
court-ordered, cross-town busing arrangements. Moreover, the burden of busing is almost always 
borne disproportionately by African American families. The second proposal bumps up against 
the reality of scarce resources for building new schools on real estate other than those parcels 
priced at the lowest end or in the least desirable sections of town. Finally, special programs 
as conceived by Justice Kennedy already exist as magnet schools. Patterns of resegregation in 
magnet schools are clear and compelling in districts like Charlotte-Mecklenburg that were once 
desegregation success stories and are now obligated to use race-neutral admissions in these choice 
programs. 


'See Capacchione v. Charlotte-Mecklenburg Schools (1999); Eisenberg v. Montgomery County Public Schools (1999); 
Tuttle v. Arlington County School Board (1999); Wessmann vy. Gittens (1998). 
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Against this backdrop of court rulings on the use of race-conscious admissions policies, in- 
creasing numbers of school districts across the country have been declared “unitary,” accompanied 
by removal of all mandatory desegregation strictures. According to the U.S. Commission on Civil 
Rights (2007), 178 school desegregation cases were dismissed after a grant of unitary status be- 
tween 1991 and 2007. Surprisingly, only limited data are available related to the aftermath of 
unitary status on the racial and socioeconomic composition of schools, or the concomitant issues 
of academic achievement, graduation rates, or students’ attitudes regarding diversity and racially 
balanced schools. Critics of unitary status predict resegregation, increased inequality, reduced 
opportunity, and a flight from diversity. Others argue that academic achievement, reduced costs 
(associated with cross-town busing), expanded parental choice, and closer neighborhood—school— 
community connections are among the benefits associated with unitary status. 

The U.S. Commission on Civil Rights (2007) examined for the first time the impact of unitary 
status on racial balance across seven states in the South, a region with a historically large number 
of districts under court order and, subsequently, unitary status.” Their report offered mixed results. 
It suggested “unitary districts have greater levels of racial concentrations than districts that remain 
under court order” (p. xii). The Commission concluded, however, that when the racial balance data 
were adjusted for the size of the districts’ student population, the percentage of White student 
enrollment, and the state in which the districts were located, the differences between unitary 
status districts and districts that remained under court order were not statistically significant. The 
report also underscored the effects of other demographic factors on increasingly diverse student 
populations, including the rapidly growing populations of Hispanic students in schools across the 
South. 

These findings are contrasted with case study research related to the impact of unitary status 
on racial integration in selected urban school districts in the South and other regions of the 
U.S. operating under grants of unitary status. Coupled with national data, these cases indicate 
growing resegregation in urban school districts (An & Gamoran, 2009; Horn & Kurlaender, 2009; 
Mickelson, Southworth, & Smith, 2009). Frankenberg, Lee, and Orfield (2003) asserted that U.S. 
schools are resegregating because of two concurrent trends: changes in the racial composition of 
neighborhoods and the end of court-ordered desegregation. Resegregation trends are evident in the 
most recent public school data (2005-06) reported by the U.S. Department of Education. These 
data indicate that Black and Latino students are slightly more isolated—or segregated—from 
White students than they were in 1993-04 (Pew Research Center, 2007). Approximately three 
out of ten minority students attended schools in 2006 in which fewer than 5% of the students were 
white (Pew Research Center, 2007). White students, in contrast, are slightly less segregated or 
less isolated from Black and Latino students than they were in 1993-2004 because of the growing 
proportion of students of color in public education generally. What explains these trends? To 
what degree are these shifts in racial concentration related to the unitary status movement? 

As the national trend toward unitary status accelerates, I examine the racial composition of 
magnet schools in a Southern urban school district declared unitary 10 years ago. This article 
is designed to inform the debate regarding the nature of unitary status environments, the use 
of voluntary choice policies such as magnet schools in these contexts, and the consequences of 
restrictions on the use of race in magnet school admission policies. 


2The U.S. Commission on Civil Rights is an independent, bipartisan agency established by Congress in 1957. 
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THE SOCIAL CONTEXT OF MAGNET SCHOOLS: HOW AND WHY 
PARENTS CHOOSE 


Patterns of parent choice are embedded in assumptions regarding the role of parents as “citizen- 
consumers” in a marketlike arrangement of school options (Chubb & Moe, 1990; Henig, 1996; 
Schneider, Teske, & Marschall, 2000; Smith & Meier, 1995). Consistent with the principles of 
rational choice theory (Elster, 1986), parents apply pressure on the demand side of the education 
market, operating under a set of ideal conditions. These include specific value preferences about 
education and schools, reliable access to adequate and accurate information about schools, a 
capacity to weigh strengths and weaknesses of various options matched to preferences, and an 
ability to choose a school that best fits parents’ preferences (Coons & Sugarman, 1978; Hamilton 
& Guin, 2005; Schneider et al., 2000). Under these ideal conditions, all parents can choose schools 
that maximize their preferences and utility.? Some studies, however, suggest that only a small 
set of parent-consumers is invested in gathering the resources necessary to make well-informed 
decisions. These “marginal consumers” demonstrate elevated will and capacity in the education 
marketplace and tend to be more highly educated and of a higher social status than the average 
parent (Meier, Polinard, & Wrinkle, 2000; Schneider et al., 2000). These attributes provide the 
currency to actually shape the market’s supply-side of school options (Schneider et al., 2000). 
Other studies suggest that parents, regardless of social class position or occupational status, 
are poor consumers of education information of the type needed to make “active and informed 
educational choices” (Ascher, Fruchter, & Berne, 1996, p. 41). 


REASONS FOR CHOICE 


Embedded in both survey and case study research, the four broad categories of parents’ rea- 
sons for choosing schools are typically clustered around academic/curriculum, discipline/safety, 
transportation/proximity/convenience, and religion/values. 

In one of the most comprehensive studies of parents’ preferences and school choice, Schneider 
et al. (2000) found that parents identify “academic” aspects of schooling as the most important 
attribute of schooling. The findings are based on surveys of 400 parents in each of four districts, 
two of which offer some forms of choice, including magnet schools or thematically focused 
alternative schools under universal choice (District 4 middle schools, New York City), controlled 
choice (Montclair, New Jersey), or option-demand choice (District 4 elementary schools, New 
York City).* The academic aspect identified most frequently is “teacher quality,” followed by 
high test scores. 


3This view, of course, has been scrutinized by claims that lower income families may have fewer material and 
social resources necessary to access reliable school information (e.g., limited knowledge of school system or few social 
networks) and may be unable to exercise choice because of financial constraints (e.g., lack of reliable transportation). 

Universal choice plans require parents to select a school in the district; there are no attendance zones or assigned 
schools and all schools are choice schools. Controlled choice refers to admissions policies that regulate parents’ choices 
to ensure that all district schools reflect a racially diverse mix of students consistent with district averages. In districts 
with option-demand plans, both choice schools such as magnets and neighborhood schools operate. Parents exercise a 
two-step process of choice: (a) choose to choose a school other than the zoned neighborhood school; (b) choose a specific 
magnet school (Schneider et al., 2000). 
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Other studies suggest that parents’ reasons for choice pivot across both push (reasons for exit) 
and pull (reasons for entry or preference) factors. This “push” explanation figures centrally in the 
reasons parents provide in a comparison study of magnet and nonmagnet school programs in St. 
Louis and Cincinnati. Dissatisfaction with neighborhood schools merge with a predictable set of 
academic reasons for choice among magnet school parents surveyed and interviewed (Smrekar & 
Goldring, 1999). Proximity from home to school also figures prominently in the reasons parents 
in these districts—particularly White parents—select schools. 


Racial Diversity (or Isolation) as a Parent Preference 


Does the racial composition of schools (and other student demographic characteristics) influence 
the patterns of parent choices? This issue gains particular currency (and urgency) in debates 
regarding the racial integration aims of magnet schools and the degree of efficacy associated 
with using voluntary choice mechanisms to achieve racially diverse schools. The research on this 
topic raises compelling questions regarding the difference between parents’ stated preferences 
and parents’ actual behavior; in other words, what parents say or report to surveyors may not 
align with what parents do (Hamilton & Guin, 2005). Thus, despite the overwhelming evidence 
regarding parents’ focus on academic quality, safety, and proximity of schools, the extant research 
on parent behavior indicates that the process of choosing schools is complex and contextually 
bound. 

Outcomes based on survey data indicate that parents are unwilling or uncomfortable identi- 
fying the racial composition of schools as a variable in their decision-making process. In this 
sense, parents may operate under assumptions of “socially desirable” responses to survey items 
that indicate values associated with social equity (Teske, Fitzpatrick, & Kaplan, 2007). A few 
studies, however, suggest that parents search for information on the Internet related to the racial 
composition of schools (Schneider & Buckley, 2002) and often act on information related to race 
in ways that indicate a preference for schools that reflect a majority population consistent with 
the parent’s racial background (Henig, 1996). 

In one of the most widely cited studies of its kind, researchers examined 450 parent transfer 
requests for admission into elementary magnet school programs in Montgomery County, Mary- 
land, in 1985 (Henig, 1990, 1996). At the time, the school district was operating a voluntary 
“managed choice” program designed to entice parents to transfer their children from neighbor- 
hood schools to thematically focused magnet schools in the predominantly minority section of 
the county. Henig (1990) found that transfer requests demonstrated a clear and consistent pattern 
across both white and minority families in the opposite direction of magnet school policy aims. 
White families were most likely to select magnet schools that were majority White and located 
in higher income neighborhoods. Minority families were also likely to seek cultural familiarity 
and racial consistency; they tended to select schools that are predominantly minority and located 
in lower income neighborhoods (Henig, 1990, 1996). 

Research in other magnet school contexts suggests that “race is a primary consideration in 
school choices” (Saporito, 2003, p. 198). Ina study that examined magnet high school applications 
in Philadelphia and compared the racial composition of neighborhood schools among Black 
families and White families who applied to magnets, the data demonstrate a clear pattern of “out- 
group avoidance” among White families; that is, Whites tended to exit neighborhood schools 
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that were majority Black through an application door to a magnet school that was less racially 
segregated (higher proportion of White students). No such pattern emerged among Black families; 
that is, application rates did not vary according to the racial composition of their neighborhood 
schools (Saporito, 2003). 

Other school choice studies, including those focused on intradistrict high schools (Saporito 
& Lareau, 1999), charter schools (Weiher & Tedin, 2002), vouchers (Schneider & Buckley, 
2002), and interdistrict elementary schools (Glazerman, 1997), find that race and class figure 
predominately in parents’ choice patterns. All point to the fact that parents tend to choose schools 
that are both academically sound (using tangible measures such as test scores and perceptions 
of teacher quality and safety) and reflect their race in the student majority (using available 
demographic information). 


SOURCES OF INFORMATION 


As noted earlier, school choice models assume that parents have access to adequate and accurate 
information on which to base their school selection decisions. The most widely cited research 
involves telephone surveys conducted with 400 parents in each of four districts, two of which 
included magnet schools or thematically focused alternative schools in New York City and 
Montclair, New Jersey (see Schneider et al., 2000). The findings indicate that information sources 
vary by parents’ income and education levels. Higher income parents and those with higher levels 
of education report greater reliance upon larger social networks; these informal networks create 
links to information-connected individuals, including professionals within the education field. 
Lower income parents and parents with lower levels of education tend to have smaller social 
networks; these networks are far less likely to include the types of resources available to higher 
income parents. Lower income parents report greater reliance upon school-based and formal 
sources of information available from individual schools, newspapers, and television. 

These differences in the size and characteristics of social networks translate into information 
deficits for lower income parents regarding the range and types of choice options available to 
them (Bell, 2005). The development and utilization of parents’ social networks are linked to 
employment status, occupation, neighborhood stability and isolation, and membership in faith- 
and community-based organizations (Cochran, 1990; Cochran & Brassard, 1979; Stanton-Salazar 
& Dornbusch, 1995). Simply stated: Who you know and where you live matters in terms of the 
size and substance of your social network. Ultimately, these networks provide the primary channel 
for collecting information about school options for children. 


CONTEXT FOR THE STUDY: NASHVILLE 


In this article, I present new evidence on Nashville, Tennessee—an important case of unitary 
status and the use of magnet schools as a tool for desegregation policy. Nashville represents 
a growing number of large urban school districts in the nation at the crossroads of sweeping 
changes in desegregation policy in the postbusing era. As a Southern city with distinctive and 
historic roles in the civil rights movement and desegregation, Nashville plays a prominent and 
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widely recognized part in the national policy debates related to school desegregation, choice, and 
equity. 

Like many Southern cities, Nashville moved slowly in response to the federal mandate to 
desegregate its schools as required by Brown v. Board of Education. For almost 30 years, the 
district used a variety of measures to stave off a full-scale desegregation. In 1983, the district 
adopted a court-approved plan that would maintain racial balance in schools throughout the 
metropolitan area. The plan included a countywide busing plan for all students beginning with 
first grade, linked to racial balance goals of 40% African American and 60% White, plus or minus 
10 percentage points. Inner-city students were bused to outlying suburban areas for elementary 
school and for Grades 7 and 8; suburban students were bused to the inner city for Grades 5 and 
6. New magnet schools were established to promote integration in the most segregated areas of 
the city. Under the plan, a student might attend as many as five different schools during his or her 
career in the Nashville schools, with the possibility of separate schools for kindergarten, Grades 
1-4, 5-6, 7-8, and 9-12. 

Frustration with continued racial achievement gaps, expensive and time-consuming bus rides, 
and lack of cohesive feeder patterns, combined with a commitment to raise standards for all 
students, led to several years of discussion and debate over student assignment patterns. In the 
fall of 1998, the Metropolitan Nashville school board and the plaintiffs in a 42-year-old federal 
school desegregation suit agreed to court-granted unitary status. Implementation was phased in 
from 1999 to 2004. 

Under unitary status, the Metropolitan Nashville Public School District adopted a $206 mil- 
lion, multiyear School Improvement Plan designed to “provide a comprehensive, high-quality 
education to every student” (Metropolitan Nashville Board of Education, 1998) while ending 
court-ordered cross-town busing in the city. The plan emphasized student assignment based on 
a “closer to home” proximity principle, coupled with a set of school choice options, including 
existing and newly established magnet schools. The expanded magnet school plan reflected the 
Board’s “commitment to providing schools with diverse enrollments because of the educational 
benefits provided in such settings” (Metropolitan Nashville Board of Education, 1998). 

During the 5-year implementation plan from 1999 to 2004, the district added five new magnet 
schools (two elementary and three middle) to the eight magnets (a mix of thematic and selective, 
program-within-school arrangements in two elementary, three middle, and three high schools) 
that existed prior to the implementation of the plan. In doing so, the Metropolitan Nashville 
school board noted a continued interest in maintaining “diverse enrollments” ... defined to 
include “racial and socio-economic and any other diversity” (Metropolitan Nashville Board of 
Education, 1998). To avoid further court action or future litigation, the Board ended the race- 
based “dual lottery” systems that had been used under court order to achieve racial balance in the 
magnet programs. After some consideration, the Board voted to reject the use of socioeconomic 
status as a diversity factor in the magnet assignment process. All the new magnets schools 
were established in predominantly African American, low-income neighborhoods, based on an 
assumption by the board that middle-class White parents from other areas would be drawn to the 
inner-city magnet schools by the array of new instructional choices. Parents would have to rely 
on their own transportation or public buses. 

In an effort to balance access and equity with expanded choice and diversity, the Board 
established “proximity priority zones” (PPZs) for the district’s magnet schools—almost all of 


which are located in predominantly African American, lower income neighborhoods in the city. 
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The proximity zones correspond to the earlier established attendance zones for schools prior to 
their switch to magnet status. After considering a set-aside for neighborhood children of 50% for 
the entry grade (K for elementary, fifth grade for middle), the Board settled upon a cap of 30% 
of seats guaranteed to the children in the priority zone (admission by lottery if proximity zone 
applications exceed the number). Magnet school applicants who reside outside of the proximity 
zone compete for the remaining 70% (or more if less than 30% of the slots are occupied by PPZ 
students) of the seats at the entry grade level in a separate lottery. 

Families in the PPZ exercise the option of selecting the magnet school in their neighborhood 
through formal application to the Magnet Schools Office. If families do not formally apply to the 
Magnet Schools Office for the PPZ lottery, they are simply assigned to the regular public school 
in their zone. The Board reasoned that this PPZ plan addresses the issues of access and equity 
for those families residing in neighborhoods impacted by new magnet schools (who, in a sense, 
“Jost” their neighborhood school to magnet status). At the same time, because 70% of the seats 
were guaranteed to families outside of the magnet school neighborhood, the goals of diversity 
and expanded choice outlined in the School Improvement Plan for the district could still be 
met. 


METHODS 


This work explores the aftermath of these policies on racial balance in Nashville’s magnet schools. 
District data (enrollment histories) are used to document the racial composition of the magnet 
schools across the city, a portrait that suggests increasing racial isolation for both White students 
and Black students. These data provide the analytical scaffolding for a qualitative case study 
of parents’ access to information, nature of social networks, and patterns of parent choice. For 
over 30 months between 2002 -04, semistructured interviews were conducted with the principals 
(current and former) and 12 to 15 teachers at each of three magnet case study schools (two 
elementary and one middle school; two newly established under unitary status) as well as with 
parents of 20 to 25 students at each school (including magnet and local neighborhood/PPZ parents) 
selected randomly from across the student population at two grade levels (elementary: third and 
fourth; middle: sixth to seventh). In addition, the former and current Directors of the Magnet 
Schools Office and the Director of the Student Assignment Office were interviewed regarding 
magnet school outreach and marketing strategies, parent choice patterns, and district enrollment 
policies. 

All interviews were audiotaped and transcribed verbatim. Interview transcripts and document 
analyses were coded and summarized according to general descriptive categories derived from 
the conceptual framework and interview protocol: knowledge and history of racial diversity 
in Nashville schools, role of magnet schools, perspectives on racial diversity and education, 
knowledge of the new school assignment plan/postunitary status, parents’ reasons for choosing, 
parents’ social networks, parents’ sources of information. Converging pieces of information 
from interview transcripts, field notes, and document analyses were arranged according to these 
categories and broad themes. All analyses were conducted using the constant comparative method 
(LeCompte & Preissle, 1993). Pattern coding (Patton, 2002; Yin, 1989) was used to discern 
patterns of thought, action, and behavior among subjects/respondents and schools. 
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TABLE 1 
Metropolitan Nashville Public Schools Student Demographics, 1992-2005 

a 
Ethnicity 9293 95-96 )=— 99-00 Ss °04-05._~—S— Davidson County Census Data 2000 

White 58% 54% 48% 40% White 65% 
Black 39% 41% 45% 46% Black 26% 
Hispanic 1% 1% 4% 10% Hispanic 5% 
Asian 2% 3% 3% 3% 2% 
Native American 0% 0% 0% <1% Native American <1% 
FRL 39% 45% 49% 63.6% Poverty rate for children 19% 


Total enrollment 68,448 68,978 68,345 70,089 younger than 5 


i NE 


Note. Metropolitan Nashville Public School District (2005); U.S. Census Bureau (2002). 


Jasper Paideia Magnet School? 


This article focuses on one of the three case study magnet schools from a larger study. Jasper was 
established before the grant of unitary status (the other two magnet case studies were established 
after the grant). It is representative in terms of size, thematic focus, location, socioeconomic 
composition, and racial balance of Nashville’s thematic magnet schools before the district was 
declared unitary. 

The Jasper Paideia Magnet School was established in 1993 as one of four “incubator” magnet 
schools in Nashville designed to create innovative curricular options for parents across the 
district. Until 1999, the district used race-based, dual lotteries to create a racially balanced student 
enrollment at Jasper and other magnet schools consistent with the court-ordered desegregation 
plan. The parents interviewed for this study enrolled their children in the kindergarten grade at 
Jasper Elementary in the first year (1999) following the grant of unitary status. The parents were 
interviewed in the spring of 2004, when their children were enrolled in fourth grade. 

Against the backdrop of increasing percentages of African American and Hispanic student 
enrollment in the Nashville district between 1992-2005 (see Table 1), the racial composition 
of Nashville’s thematic magnet schools tipped dramatically—far exceeding these demographic 
changes. As Figure | illustrates, the racial composition of the district’s magnet schools shifted 
from racially balanced under court order to increasingly racially isolated under unitary status. 
This shift occurred following the end of dual, race-based lotteries mandated under court order 
that were replaced with a single, race neutral lottery under unitary status. Jasper Elementary fits 
within this pattern (School F, Figure 1), where African American student enrollment doubled 
from 42% in 1998-99 (the last year the district operated under court order), to 84% African 
American enrollment in 2003-04, the final year of implementation of the postunitary status 
school improvement plan (see Table 2). Although most magnet schools have experienced an 
increase in the percentage of students in poverty (see Figure 2), the rates vary widely across 
schools. At Jasper Elementary, the free and reduced lunch rate increased from 35% to 47% 
between 1998 and 2004. 





> All school names are pseudonyms. 
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FIGURE 1 Percentage of Black students across thematic magnet schools, 1998-2004. Note. Metropolitan 
Nashville Public School District (2005). 


These patterns prompt the following key questions: Why did the racial composition in thematic 
magnet schools tip to predominantly African American in the aftermath of unitary status in 
Nashville? Without the “controls” of dual, race-based lotteries, what drives these results when 
districts rely on broad and targeted marketing, word-of-mouth, and single, race-neutral lotteries? 
What about the “closer-to-home” principle? Was that marketed, too? What do we know about 
parents’ social networks, school choice patterns, and value preferences that may help explain 
these new enrollment patterns across magnet schools? 





TABLE 2 
Shifts in Racial Composition at Jasper Elementary, 
1998-2004 

School Year Black Students White Students 
9899 42% 53% 
99-00 59% 38% 
00-01 T4% 23% 
01-02 719% 19% 
°02-’03 81% 16% 
03-04 84% 1% 


Note. Metropolitan Nashville Public School District (2005). 
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FIGURE 2 Percentage of free and reduced lunch students across all thematic magnet schools, 1998-2004. Note. 
Metropolitan Nashville Public School District (2005). 


FINDINGS: HOW POLICY AND SOCIAL CONTEXTS MATTER 


This section answers these questions by examining three elements—parents’ social networks, 
parents’ reasons for choice, and postunitary status policy contexts—and applies these lenses to 
an analysis of the Nashville magnet school case study. The findings suggest that these forces 
have shaped the racial composition of magnet schools within a system of choice at a moment that 
defines “a tipping point” in this urban school district. 


Social Networks and the Role of Marginal Consumers 


The first element refers to the social networks that shape parents’ information base and ulti- 
mate choices. Schneider et al. (2000) underscored the role of “marginal consumers” who create 
pressures for innovation or change in school choice arenas. These “movers” (Teske, Schnei- 
der, Mintrom, & Best, 1993) pull demand-side triggers that shape magnet school policies and 
the selection of particular instructional or curricular programs, such as Montessori or math— 
science-engineering themes. Marginal consumers (and movers) hold pivotal positions within 
social networks that enable them to activate word-of-mouth channels that help dictate the degree 
to which parents from a particular workplace, child care center, or church are informed and 
motivated to participate in a magnet school program. 

The Nashville case study findings underscore the prominent role of parents’ social networks 
and marginal consumers as a defining feature of magnet school information and school choice. In 
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the Jasper Elementary School interviews, parents and teachers alike identified preschool-based 
networks as one of the most critical channels for information sharing and decision making among 
White parents. The relationships established with these “connectors” (Gladwell, 2000) at the 
preschool were referenced repeatedly as the primary sources of information, trust, and reliable 
communication utilized in the choice-making process. For African American parents and other 
White parents uninvolved in these connector preschools, a specified set of church- and work- 
based social networks formed key information and influence portals. One African American 
parent interviewed for this study offered a keen insight that resonated across parent perspectives 
on social networks and school choice: “You don’t know schools or funeral homes if you’re not 
from here, so the folks I work with are really important that way.” 

These social networks remained stable under court-ordered desegregation plans in Nashville 
and remained in place later, under unitary status. However, a notable shift in parents’ primary 
sources of information and the social context of choice occurred under unitary status. This shift 
relates to the expanded use of Web-based networks that diminished the primacy of face-to-face 
interactions within established social networks.° With the pace and penetration of the Internet 
displacing the formerly central role of social networks, both African American and White parents 
interviewed in the Jasper case study (and in the other Nashville magnet school case studies) 
reported more insular and socially isolated social practices related to the process of school 
choice. 


Parents’ Reasons for Choice 


The second element involves the “pull” elements (Smrekar & Goldring, 1999) that parents 
identify as their “reasons for choice,” including teacher quality, safety, and school location. 
Magnet schools are built upon this principle of attraction to a particular theme (e.g., math and 
science or performing arts) or instructional approach (e.g., Paideia or Montessori). 

In the Nashville magnet school case studies, parents were drawn to conceptions of “something 
better” academically while the district was under a court-ordered desegregation plan, consistent 
with the findings reviewed earlier from magnet school studies in four urban districts (Schneider 
et al., 2000). This powerful pull, or what others refer to as the “Stickiness Factor” in consumer 
choice (Gladwell, 2000), was reflected repeatedly in parents’ perceptions of magnet schools 
as “gilded,” and “a step above public schools,” and possessing “status like a private school.” 
Although parents expressed strong views of the comparative advantage of magnet schools— 
including safety and teacher quality features—parents rarely offered any documented evidence 
or identified any measures to support these beliefs and assertions. Under conditions of cross-town 
busing patterns (preunitary status) and an option-demand choice structure, magnet schools were 
perceived to be a better alternative than regular public schools among many middle-class African 
American and White parents who were reluctant to participate in the desegregation plan that 
assigned children to schools scattered across the city—often a 45-min, one-way bus ride away 
from their home. 


©Recent research indicates a growing trend toward greater reliance on the Internet among higher income and more 
highly educated parents engaged in school choice searches and decision-making activities, making school choice a rather 
insular, individualized, and perhaps privileged activity (Buckley & Schneider, 2003; Schneider & Buckley, 2002). 
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According to district officials and magnet school application data for the years leading up to 
unitary status, the waiting lists for African American students have always been longer at Jasper 
Elementary School and in some of Nashville’s other thematic magnet schools, indicating that 
these magnet programs have always attracted a larger proportion of African American families 
than White families. The longer waiting lists for African American students were masked by the 
court order that stipulated a 60/40 majority White racial balance, plus or minus ten percentage 
points; as a result, actual enrollment figures were higher for White students. This imbalance in 
applications may reflect the demographic dispersion (neighborhood segregation) and different 
choice sets of the White and African American families in Nashville. Notably, the majority of 
residents in the inner city, where most of the magnet schools are located, are African American. 
In other words, the decision to enroll in a magnet school for many African American parents 
in Nashville represents a rational choice between a neighborhood school and a (magnet) school 
perceived to be uniformly “better,’ and for some inner-city families, the magnet school is the 
closest “neighborhood school.” Noting a pattern of uneven neighborhood school quality across 
the city and the racial imbalance at Jasper, one African American parent said succinctly, “Some 
people have better options than others.” 


Magnet Schools: Choosing Diversity? 


In addition to factors of perceived magnet school quality and cross-town busing arrangements, how 
did racial diversity imperatives shape parents’ earlier choices and how does diversity influence 
those decisions in a postunitary environment? In other words, did unitary status in Nashville dilute 
the relative significance parents attach to racial balance in schools and classrooms? There is no way 
to know whether diversity was directly related to parents’ reasons for choosing magnets when 
the district was under court orders. As noted, the evidence suggests that parents’ preferences 
were based primarily (but perhaps not exclusively) on the pull of magnet school quality and 
the push away from cross-town busing. In this postunitary status project, the case study data 
highlight the strikingly divergent viewpoints of parents on racial diversity as a reason for choosing 
schools. Notably, both African American and White parents differed on the subject; some parents 
emphasized the centrality and continuing significance of race and diversity, whereas others 
discounted the importance of racial balance and underscored the importance of safe, high-quality 
schooling, period. The following are typical of the range of parents’ perspectives expressed on 
the topic of diversity in schools, a pattern that lacked any consensus view: 


The only thing I say is that all schools should be quality schools. No one school should be better 
than the other... as long as they put the money into the schools, and put the right teachers in there, 
teachers that care about students. (parent, African American, accounting clerk) 


This is not an all black world. It never will be. . . I’m for busing. I don’t care how they get us together, 
but I think it is necessary for this world to survive.... We need to learn to understand each other in that 
we do have differences and those differences are important. (parent, African American, homemaker) 


This mix of perspectives among African American parents in Nashville was mirrored among 
White parents interviewed: 
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I don’t think I would have thought much about it (a single race school). If I were happy with the 
school and happy with how things went, then I would be happy. (parent, White, receptionist) 


Life is not a room full of Black people. She needs to learn... You cannot move through life if you 
are racially segregated in the school system... I don’t want them going to an all-white school either. 
(parent of two adopted children who are Latino; attorney, White) 


Unitary Status Policy Environment 


As magnet school application data and these interviews indicate, policy contexts matter in shaping 
parents’ choice sets and ultimately, the racial composition of magnet schools. Whether the address 
is in a predominantly White or predominantly African American neighborhood, research suggests 
that cultural familiarity and racial consistency matters in the choices parents make (Glazerman, 
1997; Henig, 1996; Saporito, 2003; Saporito & Lareau, 1999; Weiher & Tedin, 2002). 

The classroom context matters in terms of the degree to which White parents perceive the 
school to be “tipping” toward 40% or greater minority student population. As Rossell and Crain 
(1973) noted early in the magnet school and desegregation movement, this threshold can produce 
a “White flight” exit from the school, exacerbating the intent of magnet schools as a voluntary 
choice mechanism for racial diversity. 

At Jasper, the principal noted the dramatic impact of the new School Improvement Plan 
in the fall of 1999. In that first year of new “closer-to-home” student assignment plans and 
cluster schooling, the number of schools attended by any student was reduced to three (from five 
that operated under the court order). A new racial imbalance in the Nashville magnet schools, 
and at Jasper specifically, developed that first year in the kindergarten class, triggered by the 
policy context established under unitary status that ended cross-town busing patterns and shifted 
parents’ priorities and perceptions away from (cross-town) magnet schools. Many parents echoed 
the following viewpoint: 


Without busing, there is no force; they can stay in their neighborhood. And of course neighborhood 
schools have been touted as being the end all and be all, which I don’t necessarily believe. And if 
you believe in your neighborhood school, then you are going to stay in your neighborhood school. 
(parent, White, freelance writer) 


White parents who opted to drive their children across town to magnet schools (the district 
provides only limited public transportation) now had neighborhood school options. African 
American children, including those who live in lower income areas or who reside in middle class 
neighborhoods, represented an increasing majority of magnet school applicants who actually 
enrolled from 2000 through 2004, following unitary status. Magnets still presented “something 
better” to these parents compared to their neighborhood school contexts. As incentives shifted, 
the racial composition of the magnet schools tipped out of balance perceptibly with increasing 
and unbreakable momentum. Jasper teachers, Jasper parents, and potential Jasper parents all 
noticed the shifts. The principal, who is White, explained the rippling effect of these changing 
racial demographics for the school and the aim of diverse student enrollments: 
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I have already had a couple of parents come by, white parents, for next year, that were just so excited. 
And then they look around and you watch their eyes grow larger, and you can just watch it happening. 
Then they’ll say, “I don’t see a lot of diversity.” And we have to say, “no we don’t have a lot of 
diversity.” And so it is kind of a shock. ... There is nothing you can do at that point. And they will 
have heard, and they will say, “we have heard such good things about this school.” (principal) 


A teacher offered a similar experience with a parent who later disenrolled her daughter because 
of the sudden and dramatic racial imbalance at Jasper following the grant of unitary status: 


Now last year, I had all African Americans and one little white girl... this year I had one little white 
girl and her mother moved her. She told me, “it is not you, I don’t have anything against you, (my 
daughter) loves you, likes you.” But she said, “I just don’t feel comfortable with her being alone.” 
(teacher, White) 


Other powerful changes in desegregation policy tended to diminish the “pull” elements that had 
created racially balanced student enrollments in magnet schools while the district was under 
court order. Parents and teachers interviewed during the 2003-04 School Improvement Plan 
implementation period pointed to the “problem” of the district’s new PPZ policy that granted 
priority to neighborhood students in the entry grade level in the district’s thematic magnet 
schools—almost all of which are located in predominantly African American, lower income 
neighborhoods in the city. Jasper parents and teachers pointed to these PPZs as new policy 
pressures that undercut the level of parental community and social cohesion in the magnet 
programs. PPZs seemed to trigger new concerns about parents whom it was perceived—by 
non-PPZ parents—enrolled their children in magnets simply because their children were zoned 
there, rather than the PPZ parents choosing the schools affirmatively, among other options, for 
the curricular and instructional focus. Teachers worried about lack of parental understanding of 
distinctive Paideia or Montessori programs and the increasingly racial imbalance in the classroom: 


When a mother and father come and glance in a classroom on one of the visiting days to observe 
our program, for many white families to see an overwhelming number of minority students. ... I’m 
not judging them, just offers a certain response, perhaps some doubts, misgivings. “I’m not sure I 
want my child here.” Yes, partly because of stereotypical views, prejudice views about persons of 
minorities. “Well, maybe the school will be too rowdy. There won't be enough discipline,’ whatever. 
Those are things I have heard parents express. Not that they are true. They are just impressions. .. . I 
think this is important to add, I have heard any number of African American parents though, have a 
similar response when they come to the door of the classroom. (teacher, African American) 


The 40% White flight principle (Rossell & Crain, 1973) hit the tipping point as the unitary status 
plan unfolded in Nashville and new neighborhood student assignment plans were established 
across the district, reconstituting the demographic composition of schools formerly the result of 
court-ordered racial balance and cross-town busing patterns. The Jasper principal summarized 
the trend she had observed over the past 5 years under unitary status by simply stating: “As we 
tipped, it was harder and harder to get white parents, and we tipped further.” Today, the African 
American student enrollment at Jasper sits at 96% (Tennessee State Department of Education, 
2008). 
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CONCLUSION 


Shifting policy contexts related to unitary status shape magnet school choice for parents across 
urban school districts, and raise questions regarding the purpose of magnet schools in districts 
granted unitary status: Are magnet schools a viable tool for diversity under unitary status? How 
does the end of race-based magnet school admissions policies (re)shape parents’ priorities and 
choices? What are the implications for these patterns of school choice for the design, intent, and 
sustainability of magnet school programs? New legal rulings (Parents Involved in Community 
Schools y. Seattle School District) indicate that race-based magnet school admissions policies 
are impermissible, even those once considered “narrowly tailored” to achieve “compelling state 
interests” of racial diversity (Smrekar, 2007; Welner, 2006). It may not matter. The parent and 
teacher interviews in the Nashville magnet school case studies suggest the declining significance 
of racial balance (but not race, as the tipping point principle underscores) in schools, pointedly 
displaced by pressing imperatives associated with student proficiency, school improvement, and 
the legal prongs permeating teaching and learning under the federal No Child Left Behind law. 
As the plaintiff’s attorney in the original Nashville desegregation lawsuit framed the new fight: 


The primary concern I have is, are kids learning? Are our educational services being delivered? 
Ultimately, ultimately, it is up to the teachers. And all of this other stuff is just a vehicle of delivering 
services to children. So truly my particular concern is that teachers be the best that they can be. 


The new legal landscape enveloping magnet schools elevates these issues, making them all the 
more pivotal for magnet school educators, policymakers, and parents to consider within the 
framework of the original purposes and historical mandates connected to diversity aims. In 
a policy environment dominated by arguments for expanded choice, innovation, performance 
incentives, and school improvement, the purposes of magnet schools will likely be debated, 
redefined, researched, and most certainly rigorously tested. The significance of this study is 
rooted in the need for a better understanding of new policies on race and schools, the social and 
political context of choice, and the consequences of these reform strategies for school systems in 
urban America and for the lives of educators, students, and their families. Thus, the implications 
of this case study extend beyond the city limits and school corridors of Nashville, Tennessee. 
Perhaps the Jasper principal put it most pointedly: 


We are not able to teach them tolerance. And we are not able to teach them how to work with other 
people because we don’t have that to offer them. And so I hate it because I feel like we are doing a 
great injustice to the children because it is so unrealistic that they are going to work in a world like 
we have. 


This article provides a template for considering the national implications of magnet school 
policies in the era of postunitary status undergirded by embedded assumptions of expanded 
parental choice, diversity, and equity. The Nashville context underscores the ways in which 
choice and equity compete and coalesce against the background of new district priorities and 
policies on race and racial balance. 
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This article explores the relationship between charter school racial composition, school environments, 
and student achievement. We offer an original framework for understanding school context and its 
influence on schooling outcomes. We conclude that policymakers could better attend to the persistent 
educational inequality that has shaped U.S. schooling if when designing school choice plans they 
took account of student racial composition even in a postdesegregation environment. 


Charter schools: I doubled the number of charter schools in Illinois despite some reservations from 
teachers unions. I think it is important to foster competition inside the public schools. 
— Senator Barack Obama (D-IL), Third Presidential Debate, Hofstra University, October 15, 2008 


So choice and competition amongst schools is one of the key elements that’s already been proven 
in places in like New Orleans and New York City and other places, where we have charter schools, 
where we take good teachers and we reward them and promote them. 

— Senator John McCain (R-AZ), Third Presidential Debate, Hofstra University, October 15, 2008 


INTRODUCTION 


Charter schools began in Minnesota in 1991, and by 2009 they had expanded to 40 states, Puerto 
Rico, and the District of Columbia. In 2008, there were more than | million children attending 
approximately 4,000 charter schools. Although the Democratic and Republican presidential can- 
didates agreed on few policy issues during the 2008 campaign, they saw eye-to-eye on expanding 
charter schools. Yet among researchers, there is more disagreement regarding the effects charter 
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schools have had on student achievement than might be assumed from the candidates’ support of 
the charter movement. Researchers also disagree about which organizational, fiscal, curricular, or 
demographic factors contribute to charter school outcomes. These disagreements are particularly 
important as policy makers increasingly rely on charter legislation to improve U.S. schools in a 
postdesegregation era. 

Despite numerous national, state, and district-level studies, two issues remain unclear: how 
charter school outcomes fare in comparison to those of traditional public schools and how to best 
execute such comparisons. A 2004 report issued by the American Federation of Teachers (AFT) 
reinvigorated many of these debates when it found that charter schools underperformed in relation 
to traditional public schools (Nelson, Rosenberg, & Van Meter, 2004). Although advocates were 
quick to defend charter schools, most researchers agree that charter school performance has been 
mixed and that relatively few schools have been closed due to lackluster academic performance 
(Hill, 2005). There remains significant disagreement about why this is the case and what measures 
might best capture how charter schools are doing. Despite rigorous and well-intentioned efforts 
to establish consensus on how to best measure charter school student achievement and how to 
optimally compare it to performance in traditional public schools, no agreement exists in the 
research literature about methodological approaches (Lubienski, Weitzel, & Lubienski, 2009). 
Moreover, much of the charter school achievement literature has paid insufficient attention to 
the great diversity of schooling forms within the charter school and traditional public school 
sectors. 

Qualitative studies of school context can enrich the research and policy debates on charter 
school achievement, though those who want to simply ascertain if charter schools are “working” 
tend to neglect such scholarship. Although research on school contexts cannot establish causal 
relationships between school characteristics and school outcomes, it can inform and broaden our 
understanding of quantitative measures of student performance. For example, some critics of 
charter school reform worried that charter schools would become havens for White students to 
escape diversity. In fact, in many states and school districts, students of color—especially African 
American students—tend to be overrepresented in charter schools, often with students from low- 
income families (Rapp & Eckes, 2007; Wells, Holme, Lopez, & Cooper, 2000). These enrollment 
patterns hold even in districts where African American students are not the majority population, 
suggesting that some families are choosing to enroll students in charter schools in part because 
of the student racial composition (Garcia, 2008). Further, families appear to choose schools 
largely based on nonacademic considerations—safety, racial/ethnic identification, or school size 
(Schneider & Buckley, 2002; Weiher & Tedin, 2002). These demographic trends are examples 
of how issues of race, social class, and student selection complicate charter school achievement 
research and why performance analyses should be supplemented by demographic considerations 
in explanations of outcome data. 


Organization of the Article 


This article reviews the literature on charter school achievement, and then, integrating this litera- 
ture with primarily qualitative and mixed-methods studies, presents a framework for understand- 
ing broader contextual issues that impact student performance. In the next section we discuss 
three central findings that emerge from research on student achievement in charter schools. First, 
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the majority of the studies on charter school achievement rely exclusively on statistical measures, 
and although they often provide needed baseline data, they do little to explain why charter schools 
post the results they do. Second, there exists an intrasector racial achievement gap within the char- 
ter school terrain that goes underexamined in the face of cross-sector achievement comparisons. 
Third, we find that most achievement studies tend to focus on general sectors for comparisons, 
for example, “charters” are compared to “traditional public schools.” This sector-centric view 
obscures the political, social, economic, and educational dynamics within sectors and between 
schools. It also neglects the interaction between sectors—where students often move between 
charter, traditional public, and even private schools (Garcia, McIlroy, & Barber, 2008). 

These three findings informed our understanding of some of the contextual issues that help 
to explain student and school achievement. In the third section, we draw from qualitative and 
mixed-method studies to create a framework for identifying school factors that contribute to 
student performance in charter schools. This framework considers what charter schools “do,” 
what they “have,” and what they “know.” Although we do not attempt to draw causal relationships 
between these contextual factors and performance, we argue that these factors help to explain 
and interpret charter school achievement. In the article’s conclusion, we discuss the implications 
of the framework for future charter school research and policy. 


THE CHARTER SCHOOL ACHIEVEMENT DEBATES 


Charter school advocates have long claimed that charters would produce greater student achieve- 
ment for all students, especially for students of color (Nathan, 1996). Believing that traditional 
public school bureaucracy stifled educational innovations, early charter adherents were sure that 
with autonomy from school districts, charter schools’ outcomes would be superior to traditional 
public schools. 

These claims were called into question when in 2004, an AFT comparative analysis of charter, 
traditional public, and private school performance on the National Assessment of Educational 
Progress suggested that charter schools and some religious schools underperformed in math and 
reading when compared to traditional public schools (Nelson, Rosenberg, & Van Meter, 2004). 
The report also found that charter school students were less poor than public school students, 
even though charter schools enrolled a greater proportion of central city and minority children. 
The authors concluded that charter schools were not helping to reduce the racial achievement 
gap, and were also falling short of claims of improved student achievement. 

There are several reasons the AFT report was a watershed moment. The report was highly scru- 
tinized given the AFT’s historical opposition to charter school reform, the study’s methodology, 
and because it challenged the findings of many prominent school choice researcher/advocates. In 
addition, the report attracted significant media attention, including a lead story in the New York 
Times (Hendrie, 2004). A backlash quickly ensued. 

With funding from the pro-school choice Center for Educational Reform (CER), a group of 
researchers took out a full-page advertisement in the New York Times in which they denounced 
the AFT study. The advertisement questioned the methods and validity of the study and called 
for more value-added studies that assess yearly student learning gains, given achievement levels 
at enrollment. Moreover, the president of the CER, Jeanne Allen, who just 5 years earlier had 
claimed charters were clearly superior, argued that it was unfair to judge charter schools’ seeming 
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lackluster achievement because many charters served poor students and racial minorities who 
were more difficult to educate and who were starting out further behind. The controversy over the 
AFT study played out for weeks in the mainstream media, in countless blogosphere debates and 
think-tank-produced research studies (Greene, 2006). Although even those who were not charter 
school advocates questioned the methodology of the AFT study (Carnoy, Jacobsen, Mishel, & 
Rothstein, 2005), subsequent research supported many of the AFT’s study’s central findings 
(Lubienski & Lubienski, 2006; Nelson et al., 2004; U.S. Department of Education, 2004) and 
concluded that charter school achievement was mixed (Hill, 2005). 

A related set of literature reviewed charter school evaluation research and criticized many 
aspects of charter school evaluation, calling for better research design (Carnoy et al., 2005; The 
Charter School Achievement Consensus Panel, 2006). Research on charter school achievement 
that can generalize across schools and sites is complicated by partisan reports, lack of access to 
high-quality data sources, diverse state legislative environments, and schools that are different by 
design. Despite these challenges, researchers have employed several strategies to accommodate 
this complexity, which have competing strengths and weaknesses. For example, two approaches 
are single point-in-time studies and value-added studies. 


Single Point-in-Time Studies 


One of the most significant challenges to measuring performance is the likelihood that charter 
school students are fundamentally different than their traditional public school counterparts and 
from other charter school students (Zimmer & Buddin, 2005a). To deal with this, many studies 
compare charter school students to students in public schools that charter school students had 
previously attended or to students in schools that are similar to the public schools that the charter 
students would otherwise have attended. Researchers also build in controls for various student 
characteristics, including race/ethnicity, eligibility for free or reduced-price lunch, and limited 
English proficiency. 

Other single point-in-time studies have tried to account for selection bias by “matching” the 
charter schools they are examining with comparable noncharter schools. This has been done 
by matching each charter school to its neighboring public schools geographically and in terms 
of racial composition (Hoxby, 2004). It has also been done by matching charter schools to 
traditional public schools with similar characteristics—percentage of minority students, pupil 
socioeconomic status, and percentage of English language learners (Zimmer & Buddin, 2005b). 
These studies yielded different results, which have been hypothesized to be the result of method- 
ological decisions such as “the omission of more than half of the charter schools from Hoxby’s 
sample, the use of different proficiency standards for the charter and comparison schools, and 
errors in the identification of closet neighbors” (Buckley & Schneider, 2007, p. 95). Despite 
attempts to address these issues (Hoxby & Rockoff, 2005), these matching approaches continue 
to have issues of selection bias associated with them (Zimmer & Buddin, 2005a). 


Value-Added Studies 


A critique of snapshot studies is that point-in-time data fail to include a measure of students’ initial 
academic achievement as a control variable, which makes it difficult to assess the “value-added” 
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of charter schools relative to other schools (The Charter School Achievement Consensus Panel, 
2006). Many researchers agree that examining year-to-year score changes rather than single-year 
score levels yields more valid data. Consequently, researchers are increasingly using value-added 
models to evaluate the performance of charter schools. 

A study conducted by the United Teachers of New Orleans and the American Federation of 
Teachers of New Orleans (2007) traditional and charter schools after Hurricane Katrina used 
2005 and 2007 scores on the Louisiana Educational Assessment Programs to compare how 
fourth-grade students in 2007 compared to students in the same schools in 2005. According to 
this study, “10 charters improved in both subjects (reading and math), 4 improved in only one 
subject, and 16 declined in both subjects” (p. 7). Using a similar comparison strategy, Zimmer and 
Buddin (2005a) examined charter school achievement in California using “2,592,518 student- 
year observations from Los Angeles and 442,532 student-year observations from San Diego 
from 1997-1998 through 2001-2002 school years” (p. 5). Growth in achievement scores on 
the Stanford 9 Achievement Test in reading and mathematics showed that charters are not 
outperforming traditional public schools. Further, the authors found that the “charter effect” does 
not vary systematically with the race/ethnicity or English proficiency status of students, though 
Black charter school students in San Diego had statistically significant lower test scores in reading 
and math at the secondary-school level. 

Even more ambitious than comparing school-level growth is the task of tracking the growth 
of individual students. Longitudinal data on student achievement attempt to disaggregate school 
effects on student performance from student characteristics. This research has evaluated charter 
schools based on students’ own rates of annual growth before and after entering charter schools 
(Bifulco & Ladd, 2005; Booker, Filpatric, Gronberg, & Jansen, 2004; Gronberg & Jansen, 2001; 
Hanushek, Kain, Rivkin, & Branch, 2005; Sass, 2005; Solmon, Park, & Garcia, 2001; Zimmer 
et al., 2003). 

As with other charter school performance studies, the findings of value-added analyses 
have been mixed. The national charter school network that operates under the Knowledge 
is Power Program (KIPP) has received significant media and researcher attention and philan- 
thropic support because of their high student achievement scores as well as their success with 
transitioning students from middle school to elite public and private high schools. Such re- 
sults have led policymakers to conclude that the KIPP model of longer school days and years, 
parent and student contracts, emphasis on discipline, and structured curricula should be repli- 
cated across urban systems (Thernstrom & Thernstrom, 2003). Research on KIPP, however, 
has had difficulty establishing the educational value KIPP charter schools provide their stu- 
dents. One study compared student achievement outcomes of three KIPP Diamond Academies 
and their matched control counterparts (Gallagher & Ross, 2005) by looking at the difference 
between “pretest” scores and scores on the 2004 Tennessee Comprehensive Assessment Pro- 
gram/Achievement Test. When charter and non-charter students with similar initial scores were 
compared, findings showed significant advantages for the fifth-grade cohort and the sixth-grade 
cohort who had attended KIPP for 2 years, but no advantage for sixth graders in their first year 
at KIPP. 

Two growth studies of Texas charter school students also yielded disparate outcomes. Hanushek 
et al. (2005) followed individuals for 4 consecutive years to compare average test score gains of 
charter students with the same students’ gains in district schools. They found that students’ gains 
were initially lower in charter schools but showed no significant differences after 2 or 3 years 


232 J. SCOTT AND A. VILLAVICENCIO 


of charter school life. Using similar value-added and matched sample approaches, Gronberg and 
Jansen (2005) tracked individual student performance on the state assessments (Texas Assessment 
of Knowledge and Skills) over 2 years. They found that attending a charter school for several years 
resulted in achievement gains for elementary and middle school students, whereas academic gains 
for high school charter school students was significantly lower than those attending traditional 
public schools. The study also showed that charter schools were more effective with lower 
achieving students. 


A Methodological Consensus? 


Currently, policy scholars and educational economists are trying to establish consensus about 
how to best measure charter school achievement. The Charter School Achievement Consensus 
Panel (2006) favors experimental methods, where students are randomly assigned to schools, but 
recognizes that this method is often not possible. Although the panel’s work has the potential to 
generate common standards for evaluation research on achievement, it also assumes that other 
researchers will abide by its recommendations. Of importance, the panel calls for the inclusion 
of qualitative data to understand variation in charter school achievement. One type of variation 
that is often overlooked by the previously described achievement studies is the intrasector charter 
school achievement gap. 


Charter School Segregation and the Intrasector Racial Achievement Gap 


A closer look across the research reveals that a similar racial achievement gap to the one in the 
traditional public sector is emerging within the charter school sector (Nelson et al., 2004). For 
example, White charter school students outperform Black students by roughly 20 points in math 
and reading and Latino charter students by 15 points (Carnoy et al., 2005). These data require 
close attention, especially considering that many charter schools underserve special education 
students and English language learners, and may enroll students who, although coming from 
poverty, are relatively more motivated and privileged than students in nearby traditional public 
schools (Buckley & Schneider, 2007). 

Researchers have found that charter schools are more likely to be located in urban school 
districts and enroll disproportionately more African American students than their public school 
counterparts, but enroll fewer Latino, language minority, and special education students (Welner 
& Howe, 2005). Charter school student composition is the result of several dynamics at both the 
macro policy level and within local school practices. Some indicate that families of color are 
self-sorting into racially homogeneous schools. Others find that charter school recruitment and 
marketing strategies are geared toward attracting particular demographic profiles, whereas other 
research documents the practice of charter schools to serve “niche” populations (Lacireno-Paquet, 
Holyoke, Moser, & Henig, 2002; National Charter School Research Project, 2007; Smith, 2001; 
Wells, Lopez, Scott, & Jellison-Holme, 1999). 

Another factor shaping student racial composition in charter schools is the absence of mean- 
ingful incentives or enforced regulations to create or maintain diverse charter schools. Given the 
deregulatory impetus of early charter school reformers, some states’ charter school legislation 
is either vague or silent on racial balance imperatives (Wells, Holme, Lopez, & Cooper, 2000). 
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And where more specific racial balance provisions exist, researchers have found little evidence 
that state or local authorities have the capacity or will to enforce existing guidelines (UCLA 
Charter School Study, 1998). In fact, influential policy advocates, such as CER, express little 
concern for racial balance or racial integration. Indeed, CER’s 10-point rubric for evaluating a 
“strong” charter school law does not include any mention of race or civil rights protections (Scott 
& Barber, 2002). 

Aggregate numbers of charter school racial enrollments can obscure the segregation found 
within local charter schools. Although African American students are well represented in the 
aggregate charter school enrollment, they tend to go to schools that are majority African American. 
For example, 70% of Black charter school students attend hypersegregated schools compared 
to 34% Black students in traditional public schools (Frankenberg & Lee, 2003). These numbers 
become especially stark when assessing how they play out in individual charter schools: 


Charter schools serve larger shares of African American and Latino students than their respective 
proportions found in regular public schools. But ethnic segregation is comparatively greater in charter 
schools. Three-fourths of all black charter school students are enrolled in 273 schools. The share 
of students who are African American in these schools averages 80%, compared to 54% black 
representation among the comparable set of regular public schools. (Fuller, Gawlik, Gonzalez, Park, 
& Gibbings, 2003, p. 3) 


As a result, prominent advocates now claim that charters should be given different account- 
ability consideration even when they underperform because they serve a disproportionate number 
of high-poverty students of color. After Ohio ranked most of its urban charters with failing or 
mediocre grades in 2007, for example, Jeanne Allen wrote a letter to the editor of the New York 
Times that called for a different standard for evaluating charters, and argued against closing down 
struggling schools. 


Certainly some of Ohio’s charter schools are not performing as well as had been hoped for when 
they were founded. But in many of these cases it’s because the schools have taken on the challenge 
of educating the difficult-to-reach children who were given up on by traditional public schools—the 
children who, every year, fell further and further behind and received no help; the children who, were 
it not for their charter school, would have dropped out or landed in jail or worse. For them, charter 
schools are their last best hope for receiving an education and ultimately succeeding in life. Will these 
students be better off if their charter schools go out of business? The answer must be a resounding 
no. (Allen, 2007) 


This logic about charter school achievement being singularly tied to the students them- 
selves challenges many of the central policy assumptions that originally drove charter school 
reform, including the belief that regulation caused underachievement in traditional public schools 
and that bureaucratic structures stymied educational innovation and experimentation that could 
lead to more dynamic student learning. This was claimed to be especially true in large, urban 
systems, which tend to serve poor students, students of color, and English language learners 
(UCLA Charter School Study, 1998; Wells, Grutzik, Carnochan, Slayton, & Vasudeva, 1999). 
Under the more recent charter school accountability rhetoric, the value of being a charter be- 
comes less important than school context and school practices, and multiple measures of suc- 
cess are trumpeted. Although these arguments signal a shift in how advocates are thinking 
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about charter schools and achievement, perhaps more important, it redirects research atten- 
tion to school-level factors that shape such achievement—the contexts under which students 
learn. 

We conclude from this review of charter school achievement that some charter schools have 
unique combinations of qualities that make them more likely to be high achieving, and that 
although these characteristics can be similar across charter and traditional school sectors, the 
unique policy environment of charter schools make them especially important to scrutinize in 
relationship to school performance. Charter schools are different from one another by design. “By 
design, that ‘movement’ is a collection of unique schools ranging from international baccalaureate 
academies to intensive last resorts for juvenile lawbreakers; taking their average temperature 
probably wouldn’t be enlightening” (Schorr, 2000, p. 20). Understanding how charters produce 
their outcomes can inform explanations of student achievement, as well as inform policymakers 
about whether to make public investments in supporting their practices. We next offer a framework 
of contextual issues, generated from our review of charter school research that helps to unpack 
the school-level dynamics that may explain school outcomes. 


SCHOOL-LEVEL CONTEXT: LOOKING INSIDE THE EDUCATIONAL 
“BLACK BOX” OF CHARTER SCHOOLS 


There is limited understanding about the ways in which the local charter school context, especially 
with regards to school racial composition, might illuminate the seeming underperformance of 
charter schools and explain the intrasector racial achievement gap. In this section, we bring policy 
studies and sociological literature on local charter school contexts to bear on the achievement 
issues raised in the previous section. These studies highlight the importance of student racial and 
social class composition and site-level resources—including quality teachers, facilities, funding, 
class size, curriculum, and instructional materials. These factors comprise a school’s educational 
environment, whose dynamics are often called the “black box” of schooling given the lack of 
understanding about how they directly impact schooling outcomes (Zimmer & Buddin, 2005b). 

Researchers and charter school advocates have not given sufficient attention to the qualitative 
educational environments within charter schools and the ways in which they might contribute 
to our understanding of charter school achievement data. Rather, much of the literature has 
tried to draw quantitative correlations between school-level characteristics (such as management 
and governance type) and student achievement (Peterson, 2007; Zimmer & Buddin, 2005a). 
Our framework holds that school context must be considered to help explain charter school 
achievement, especially given the lack of consensus on charter school effects. We take a descriptive 
rather than a causal approach here, in part to present options that other researchers might consider 
in mixed-methods study of charter school achievement, and because there is not sufficient study 
on these elements to support causal arguments. 

We base our framework primarily on qualitative case studies, which allow for rich and thick 
descriptions. Our review yielded three categories of school-level environments that help to explain 
the previously documented variation in student achievement. Briefly stated, our framework of 
charter school context considers what schools (a) do, (b) have, and (c) know. We summarize these 
areas in Table 1 before discussing them each in greater detail. 
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TABLE 1 
A Framework for Explaining Charter School Outcomes 


a 


Do Have Know 
ae ee ee eR SE Gest ls ewe ete wt et ares Dt ee ed 
Curriculum and teaching (specialized Private funding: Foundations, Pedagogy: Traditional, 
focus, scripted curriculum/direct businesses, individual donors back-to-basics, constructivist, 
instruction, ethnocentric) culturally relevant 
Admissions Public funding Business/fiscal/legal expertise 
Discipline policy Political/Social networks Assessment and accountability 
Student recruitment In-kind resources Fund-raising 
Personnel: recruitment, compensation, Partnerships: Universities, Marketing/branding 
retention corporations 
Governance EMO/CMO management Cultural/racial competence 
School organization: Grade levels, Student demographics: Race, Teacher and leadership 
school and class size, school day & poverty, prior academic professional development 
year, teacher-pupil ratio achievement, English language 


attainment, immigration status, 
special education 


Within school resource allocations, Teacher characteristics: Level of Community engagement 
student and teacher assignment education, certification, years of 
experience 


a 


Note. EMO/CMO = educational management organization/charter management organization. 


Examining the Practices of Charter Schools 


The first area of charter school context is what they “do”; the local practices in several areas: 
school size, curriculum and pedagogical approaches, admissions and discipline policies, person- 
nel policies, and governance. In all of these areas, charter school research reveals a range of 
practices. 

Two areas in which charter schools are generally allowed freedom are school size and school 
mission/curriculum. In terms of school size, charter schools tend to be smaller than traditional 
public schools, for example, but the average size of charter schools that are operated by educational 
management organizations (EMOs)' tend to be larger than other charter schools (Miron & Nelson, 
2002; Molnar, Garcia, Miron, & Berry, 2007). In this regard, many non-EMO affiliated charter 
schools have much in common with the small schools movement—founders, teachers, and parents 
seem attracted to schools that are small, but that also have a specific curricular orientation or 
mission (DiMartino, 2007). A focus on curricular approach or school mission often accompanies 
many charter schools, regardless of size, but within smaller schools, staffs may find it easier to 
adhere to early purposes and foci of the schools since there are potentially fewer voices of dissent 
(Wohlstetter & Griffin, 1998). When EMOs manage charter schools, however, the organization 
often brings a prescribed curriculum or pedagogical approach—leaving little room for parents 





'BMOs provide a range of governance and management services to schools on both for-profit and nonprofit bases. 
Approximately 13% of charter schools are managed by an EMO, but in particular states and districts, the concentration 
is much higher; in Michigan nearly 75% of charters are run by EMOs (Miron & Nelson, 2002; National Charter School 


Research Project, 2007). 
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or teachers to alter the approach but allowing EMOs the ability to ensure fidelity to their school 
models (Bulkley, 2005; Scott & DiMartino, 2008). 

Charter schools also have the ability to limit student enrollments, provided they adhere to 
state and federal civil rights guidelines. Formal screening mechanisms used by charter schools 
to admit students vary, however, charter schools do pay attention to which students they enroll 
and use a range of strategies to shape which students are admitted to schools, and which students 
are expulsed (Becker, Nakagawa, & Corwin, 1997; Lacireno-Paquet et al., 2002; Urietta, 2006; 
Yancey, 2000). These techniques include the use of applications, contracts, steering families away 
from enrolling, recruiting families from particular neighborhoods, school discipline policies, and 
various forms of branding to communicate to parents which students are good “fits” for the 
particular charter school (Lopez, Wells, & Holme, 2002). This attention to student composition 
is not only tied to the need to produce greater student achievement but also connected to many 
school founder’s desires to serve particular populations—often poor, African American and 
Latino students, but also “gifted” students, special education students, or students interested in 
a particular career or academic subject. Many charter schools serving such populations employ 
selective admissions processes, such as that used by the highly regarded KIPP schools.” Although 
schools in the KIPP charter network appear to generate significantly higher student achievement 
than surrounding public schools, they also require an admissions application and aparent and 
student contract, and they lose many students unable or unwilling to abide by its policies. 
Looking only at KIPP or similar schools that are racially homogenous but distinctive in other 
ways can distort research conclusions about student outcomes. 

Another area in which charter schools utilize different strategies is school personnel. De- 
pending upon state policy, charter schools have had diverse orientations to hiring noncertified 
teachers, for example. They have also differed on whether teachers can be represented by teachers 
unions. Researchers have also found that differential compensation abounds within the charter 
school sector. Although some charter schools pay teachers more than traditional public schools, 
charter school teachers might also be required to work longer school days and years (Johnson & 
Landon, 2000). Perhaps as a result of such additional work requirements, but also because of the 
intensive nature of start-up organizations, many charter schools have high teacher turnover. In 
the early years of many charter schools, entire staffs changed (Fitzgerald, 2000; Henig, Holyoke, 
Lacireno-Paquet, & Moser, 2001; Leonard, 2002; National Charter School Research Project & 
Center on Reinventing Public Education, 2007). Finally, for the most part, charter school leaders 
are able to enjoy autonomy in hiring teachers, having more control and input than their traditional 
school counterparts (Finnigan, 2007). 

Charter schools also employ different governance mechanisms. Community-based or grass- 
roots charter schools generally have trustees or boards whose members include teachers, school 
leaders, parents, and community members. EMO-run, and other charter schools have governance 
models that have less community and school-based participants and rely instead primarily on 
members of the fiscal, legal, and business communities (Rhim, 1998). Where the former gover- 
nance model tends to focus on internal school issues, the latter attends to local school issues while 
also working to generate political and fiscal support for the schools. Researchers have found that 
board members in these environments expect to have an active role in site-based decisions, which 


, : ae : 
“Charter school advocates correctly point out that there are traditional public schools that have selective admissions. 
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can sometimes result in tension between boards and school leaders and teachers (Ascher et al., 
2001; Deal & Hentschke, 2004). 


Examining the Resources of Charter Schools 


The second area is what charter schools “have.” This includes their combinations of public and 
private funding, the former of which can be of different levels, even within the same school 
district, and the latter of which can be quite significant. For example, the Broad Foundation and 
the New Schools Venture Fund have distributed millions of dollars to several EMOs over the last 
several years. 

Political and social networks can assist with building community and state support for the 
school. For example, charter schools that are resource rich have access to social, political, and 
financial networks that help to sustain their schools. These resources can help them develop 
instructional, governance, and school organizational practices that can encourage a more focused 
effort on student learning. In-kind resources such as time and expertise from parents within 
the school help sustain the school’s operations. Finally, having resources to try out innovative 
pedagogical, governance, and organizational approaches can help to build institutional knowledge 
and capacity for improved or sustained high performance. 

Partnerships with universities and businesses can also play a key role. For example, they 
can assure long-term support for a variety of measures, and provide access to internships and 
employment for students (“Mass-Producing Excellence,” 2005). Another resource is the presence 
of a management organization. Depending on the organization’s level of expertise, the school 
can enjoy the benefit of having its operations overseen by those with administrative capabilities 
(Bulkley & Fusarelli, 2007; Bulkley, Mundell, & Riffer, 2004; Burch, 2006). 

Students, teachers, parents, and community members are also resources for charter schools. 
The demographics of the students are an especially overlooked resource. As mentioned earlier, 
the kinds of students charter schools serve can not only increase their achievement results but 
also connect the school to donors looking to support particular kinds of students (Scott & Holme, 
2002). 

Finally, teachers are perhaps one of the most important resources. Their levels of educa- 
tion, certification, and years of experience are important—though not the only—indicators of 
teacher quality (Miron & Nelson, 2002). Researchers have found that teachers in racially seg- 
regated, low-income charter schools tend to have less experience and fewer credentials, and 
tend to leave schools at higher rates than their traditional public school counterparts. Although 
this turnover can help to keep salary expenditures low, it also suggests that these schools may 
struggle to build a network of teachers who can mentor and support newcomers to the school. 
Unfortunately, the charters most impacted by this trend are those comprised mainly of students 
of color (Fuller, Gawlik, Gonzalez, Park, & Gibbings, 2003). Further, charter school teachers 
in general tend to be less experienced and have fewer credentials, but this trend is magnified 
in high-minority schools. Case studies of charter schools reveal these trends in teacher qualifi- 
cations correlate with student demographics—wealthier schools tend to have more experienced 
and credentialed teachers (Fuller et al., 2003; Henig et al., 2001; Wells, 2002; Wexler & Huerta, 
2000). The context of resources, segregation, and achievement is complex within the char- 
ter school movement. There have been a number of seeming success stories (high-achieving 
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charter schools that are attended by low-income students of color). Many of these schools are 
run by EMOs like KIPP, Green Dot Public Schools, or Uncommon Schools—organizations that 
have received millions from charter supporting philanthropies and enjoy tremendous financial 
and political support from the private sector (Matthews, 2007). At the other end of the char- 
ter school demographic spectrum, students from wealthy backgrounds can bring families who 
would provide fiscal support to the school, whereas a concentration of poor students of color 
increasingly attracts donors looking to expand the choice movement to those communities (Scott, 
2008). 

Racially segregated, well-resourced, and high-performing schools tend to be the exception 
rather than the rule, however, and are examples of how school context relates to achievement given 
their selective admissions, charismatic leaders, and significant financial and political support. 
For the majority of segregated charter schools, student racial composition relates to resources, 
teacher quality, and ultimately school performance. These findings imply that being a charter 
alone is an insufficient mechanism for closing the racial achievement gap. Further, a myopic 
policy focus on charters may actually widen the gap by creating a sector of schools that are 
selective but do not necessarily have the capacity to provide students with greater opportunities 
to achieve. At the other end of possibility, we see the emergence of highly selective, racially 
homogenous charter schools that receive significant private financial and political support and 
whose outcomes are touted as exemplars for charter schools and traditional public schools alike. 
The larger racial context of schools—particularly poverty, inequality, and segregation—has been 
missing from recent educational debates. Increasingly, the public argument is that equality can 
be better achieved through greater privatization, marketization, and competition among schools 
(Wells, Holme, Revilla, & Atanda, 2005). Examining these resources helps to temper the claims 
of some charter schools’ success in closing the racial achievement gap and in the general ability 
of charter models to be replicated. In 2007, KIPP, for example, terminated its contract with the 
KIPP Sankofa Charter School in Buffalo, New York, indicating that its model does not work in 
all settings (Matthews, 2007). 


Examining the Capacities of Charter Schools 


The last school level contextual factor is perhaps the most nebulous—what charter schools 
“know.” Here we refer to the capacity of the leaders, board members, and teachers to deliver and 
to sustain high-quality schooling. 

Under the category of school knowledge, we include the pedagogical capacity, the school’s 
understanding of the particular instructional needs of its students, and the ability to meet them 
through rigorous curricula. Another area related to pedagogical capacity is that of assessment and 
accountability skills—how schools are able to use data to inform practice (Darling-Hammond, 
1997). One strategy that has been utilized by charter schools seeking to build curricular and ped- 
agogical capacity has been the establishment of partnerships with organizations who might have 
strengths in these areas (Wohlstetter, Malloy, Smith, & Hentschke, 2004). It has been common 
for charter schools to align with universities, museums, community-based organizations, and 
even corporations based on the particular needs of the schools. These alliances and partnerships 
happen under formal and informal arrangements, can bring much needed expertise to a charter 
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school, but can also raise tensions about the level of control each entity exercises over critical 
school processes (Lin & Hassel, 2001; Minow, 2003). 

Also important in the area of capacity is a combination of business, fiscal, and legal expertise 
(UCLA Charter School Study, 1998). This is especially critical for schools that are start-ups and 
have fewer ties to school district resources as well as schools partnered with EMOs or external 
educational providers. Charter school stakeholders need to be able to negotiate and oversee 
multiple contracts, for example. Charter schools can also benefit from knowledge of marketing, 
branding, and fund-raising, especially because charters are often ineligible for an equal level of 
state resources. Marketing and branding knowledge is related to fund-raising; understanding how 
to position the charter school as unique and worth attention from funders and parents alike is a 
key area of capacity (Lubienski, 2007). 

Research from school reform and school restructuring also indicates that the degree to which 
the school staff is competent to engage with the racial and cultural backgrounds of their students 
is also important (Carter, 2005; Ladson-Billings, 1995). Consistent professional development, 
focused on teachers’ weaknesses, is another issue of school knowledge. Finally, the charter 
school’s knowledge of the community it serves, and its ability to be engaged with that community, 
can have implications for the school’s longevity and success (Lipman, 1998). 

We argue that it is important to consider these three areas—what schools do, have, and know— 
when interpreting claims of charter school success or failure. This can shift current policy attention 
away from what are ultimately narrow measures of schooling outcomes and push quantitative 
and qualitative researchers to develop compelling explanations for how particular charter schools 
accomplish their results. This can inform what public and private investment, social policy, 
and oversight are required to encourage the growth of high-performing and equitable schools. 
These three areas of qualitative indicators, then, complement statistical analyses of charter school 
achievement to help explain why schools’ performance on assessments can be so varied. 


CONCLUSION AND IMPLICATIONS 


This article explored the relationship between charter school contexts and student achievement. 
Our review of the literature on charter school outcomes and our framework for explaining and 
interpreting those outcomes raise further challenges for researchers. In particular, this review 
makes clear that we need more complex and interactive methodologies to make summary conclu- 
sions about how all schools perform given their unique institutional, political, racial, and social 
environments. 

As we discussed, many charter schools have self-selected populations, exercise control over 
student admissions, and are designed to specialize in particular curricular foci or organizational 
structures, making each school distinct. Aggregating charter schools together by sector, and then 
drawing conclusions about their average performance is as unfair as evaluating traditional public 
schools together. 

The unresolved debates about charter school achievement suggest that researchers should use 
more nuanced and holistic methods than are traditionally employed. Mixed-method studies could 
paint a more complete picture of the characteristics that encourage greater student learning and 
achievement. Mixed-method studies could also pinpoint the bureaucratic, curricular, organiza- 
tional, institutional forms and resources that help support schools that are higher achieving. 
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From a political advocacy standpoint, we see at least two trends emerging that require further 
exploration: (a) an argument that charters serve harder to educate students, and so low achievement 
is to be expected, at least for a while, and (b) a shift in the accountability rhetoric that says 
charters might serve other social purposes besides increasing achievement. These claims suggest 
different purposes than originally envisaged for charter school reform. They are also particularly 
worrisome given the decrease in potential policy tools resulting from recent Supreme Court 
decisions about the use of race in achieving integrated schools. Charter schools may be one of 
the few politically viable tools left for improving the outcomes of traditionally disadvantaged 
students. Despite mixed achievement findings and little consensus on how to best measure charter 
school achievement, policymakers at all levels continue to tout charter schools as solutions for 
lackluster academic performance. 

To the extent that the United States will increase its investment in charter schools through 
a reauthorization of current No Child Left Behind provisions, further public deliberation about 
what social and educational value they hold in American public education is required. There 
are many possibilities that charter schools can bring to public education, including expanded 
choices, the opportunity to experiment and innovate, and the ability to shape more responsive 
and challenging schooling. To realize these possibilities, however, charter schools—much like 
traditional public schools—need support from policymakers to craft educational environments 
that are diverse, highly resourced, and of high academic quality. 


REFERENCES 


Allen, J. (2007, November 15). Ohio’s charter schools [Letter to the editor]. New York Times. 

Ascher, C., Echazarreta, J., Jacobowitz, R., McBride, Y., Troy, T., & Wamba, N. (2001). Going charter: New models of 
support. New York: Institute for Education and Social Policy. New York University. 

Becker, H. J., Nakagawa, K., & Corwin, R. (1997). Parental involvement contracts in California charter schools: Strategy 
for educational improvement or method of exclusion? Teachers College Record, 98(3), 512-536. 

Bifulco, R., & Ladd, H. (2005). The impact of charter schools on student achievement: Evidence from North Carolina. 
Education Finance Policy, 1(1), 50-89. 

Booker, K., Filpatric, S., Gronberg, T. J., & Jansen, D. W. (2004). Charter school performance in Texas (Private Enterprise 
Research Center Working Paper). College Station: Texas A&M University. 

Buckley, J., & Schneider, M. (2007). Charter schools: Hope or hype? Princeton, NJ: Princeton University Press. 

Bulkley, K. (2005). Losing voice? Educational management organizations and charter schools’ educational programs. 
Educationa and Urban Society, 37(2), 204-234. 

Bulkley, K., & Fusarelli, L. (Eds.). (2007). The politics of privatization in education. Thousand Oaks, CA: Corwin. 

Bulkley, K., Mundell, L., & Riffer, M. (2004). Contracting out schools: The first year of the Philadelphia diverse provider 
model. Philadelphia: Research for Action. 

Burch, P. E. (2006). The new educational privatization: Educational contracting and high stakes accountability. Teachers 
College Record, 108(12), 2582-2610. 

Carnoy, M., Jacobsen, R., Mishel, L., & Rothstein, R. (2005). The charter school dust up: Examining the evidence on 
enrollment and achievement. New York: Teachers College Press. 

Carter, P. (2005). Keepin’ it real. New York: Oxford University Press. 

Darling-Hammond, L. (1997). The right to learn: A blueprint for creating schools that work. San Francisco: Jossey-Bass. 

Deal, T., & Hentschke, G. (2004). Adventures of charter school creators: Leading from the ground up. Lanham, MD: 
Scarecrow Education. 

DiMartino, C. (2007, November). Building a marketplace: Privatization and the new small schools movement. Paper 
presented at the Annual Meeting of the University Council of Education Administration, Washington, D.C. 

Finnigan, K. (2007). Charter school autonomy: The mismatch between theory and practice. Educational Policy, 21(3), 
503-526. 


SCHOOL CONTEXT AND CHARTER SCHOOL ACHIEVEMENT 241 


Fitzgerald, J. (2000). 1998-1999 Colorado Charter Schools Evaluation Study: The characteristics, status and performance 
record of Colorado charter schools. Denver: Colorado Department of Education. 

Frankenberg, E., & Lee, C. (2003). Charter schools and race: A lost opportunity for integrated education. Cambridge, 
MA: The Civil Rights Project at Harvard University. 

Fuller, B., Gawlik, M., Gonzalez, E. K., Park, S., & Gibbings, G. (2003). Charter schools and inequality: National 
disparities in funding, teacher quality, and student support (No. Working Paper Series 03-2). Berkeley: Policy 
Analysis for California Education. 

Gallagher, B. M., & Ross, S. M. (2005). Analysis of Year 2 (2003-2004) Student achievement outcomes for the Memphis 
KIPP Diamond Academy. Center for Research in Educational Policy, University of Memphis, Memphis, TN. 

Garcia, D. R. (2008). The impact of school choice on racial segregation in charter schools. Educational Policy, 22(6), 
805-829. 

Garcia, D. R., Mcllroy, L., & Barber, R. T. (2008). Starting behind: A comparative analysis of the academic standing of 
students entering charter schools. Social Science Quarterly, 89(1), 199-216. 

Goldhaber, D. D., & Eide, E. R. (2003). Methodological thoughts on measuring the impact of private sector competition 
on the educational marketplace. Educational Evaluation and Policy Analysis, 25(2), 217-232. 

Greene, J. (2006). Apples to apple: An evaluation of charter schools serving general student populations (Ed- 
ucation Working Paper Archives). Retrieved June 2, 2006, from http://www.uark.edu/ua/der/EWPA/Approved/ 
apples_to_apples.html 

Gronberg, T. J., & Jansen, D. W. (2001). Navigating newly chartered waters: An analysis of Texas charter school 
performance. San Antonio and Austin: Texas Public Policy Foundation. 

Hanushek, E. A., Kain, J. F, Rivkin, S. G., & Branch, G. F. (2005, March). Charter school quality and parental 
decision making with school choice (NBER Working Paper 11252). Retrieved June 28, 2007, from http://www. 
nber.org/papers/w 11252 

Hendrie, C. (2004). NAEP study fuels debate over charter schools [Electronic version]. Education Week. Retrieved 
September 4, 2008, from http://www.edweek.org/login. html ?source=http%3A 7%2F%2F 

Henig, J. R., Holyoke, T. T., Lacireno-Paquet, N., & Moser, M. (2001). Growing pains: An evaluation of charter schools 
in the District of Columbia; 1999-2000. Washington, DC: The Center for Washington Area Studies, The George 
Washington University. 

Hill, P. T. (2005). Assessing achievement in charter schools. In R. Lake & P. Hill (Eds.), Hopes, fears, and reality: A 
balanced look at American charter schools in 2005 (pp. 21-32). Seattle: National Charter School Research Project, 
Center on Reinventing Public Education, University of Washington. 

Hoxby, C. M. (2004). A straightforward comparison of charter schools and regular public schools in the United 
States (HIER Working Paper). Retrieved June 28, 2007, from http://post.economics.harvard.edu/faculty/hoxby/ 
papers/charters_040909.pdf 

Hoxby, C. M., & Rockoff, J. E. (2005, Fall). Findings from the city of big shoulders. Education Next, pp. 52-58. 

Johnson, S. M., & Landon, J. (2000). Sometimes bureaucracy has its charms: The working conditions of teachers in 
deregulated schools. Teachers College Record, 102(1), 85-124. 

Lacireno-Paquet, N., Holyoke, T. T., Moser, M., & Henig, J. R. (2002). Creaming versus cropping: Charter schoool 
enrollment practices in response to market incentives. Educational Evaluation and Policy Analysis, 24(2), 145-158. 

Ladson-Billings, G. (1995). The dreamkeepers: Successful teachers of African American children. San Francisco: Jossey- 
Bass. 

Leonard, J. (2002). The case of the first-year charter school. Urban Education, 37(2), 219-240. 

Lin, M., & Hassel, B. (2001). Charting a clear course: A resource guide for building successful partnerships between 

_ charter schools and school management organizations. Washington, DC: Charter Friends National Network. 

Lipman, P. (1998). Race, class, and social power in school restructuring. Albany, NY: SUNY Press. 

Lopez, A., Wells, A. S., & Holme, J. J. (2002). Creating charter school communities: Identity building, diversity, and 
selectivity. In A. S. Wells (Ed.), Where charter school policy fails: The problems of accountability and equity (pp. 
129-158). New York: Teachers College Press. 

Lubienski, C. (2007). Marketing schools: Consumer goods and competitive incentives for consumer information. Educa- 
tion and Urban Society, 401(1), 118-141. 

Lubienski, C., & Lubienski, S. T. (2006). Charter, private, public schools and academic achievement: New evidence 
from NAEP mathematics data. New York: National Center for the Study of Privatization in Education, Columbia 
University. Retrieved February 1, 2006, from http://www.nespe.org 


242 J. SCOTT AND A. VILLAVICENCIO 


Lubienski, C., Weitzel, P., & Lubienski, S. T. (2009). Is there a “consensus” on school choice and achievement? Advocacy 
research and the emerging political economy of knowledge production. Educational Policy, 23(1), 161-193. 

Mass-producing excellence. (2005). Philanthropy Magazine. Retrieved October 1, 2007, from http://prt. 
timberlakepublishing.com/article.asp?article=747 &paper=0&cat=150 

Matthews, J. (2007). Looking at KIPP, cooly and carefully [Electronic version]. Washingtonpost.com. Retrieved February 
29, 2008, from http://www.washingtonpost.com/wp-dyn/content/article/2007/04/24/207. 

Minow, M. (2003). Public and private partnerships: Accounting for the new religion. Harvard Law Review, 116(5), 
1229-1270. 

Miron, G., & Nelson, C. (2002). What’s public about charter schools? Lessons learned about choice and accountability. 
Thousand Oaks, CA: Corwin Press. 

Molnar, A., Garcia, D. R., Miron, G., & Berry, S. (2007). Profiles of for-profit management organizations: Ninth annual 
report 2006-2007. Tempe: Commercialism in Education Research Unit, Education Policy Studies Laboratory, Arizona 
State University. 

Nathan, J. (1996). Charter schools: Creating hope and opportunity for American education. San Francisco, CA: Jossey- 
Bass. 

National Charter School Research Project. (2007). Quantity counts: The growth of charter school management organi- 
zations. Seattle: Center on Reinventing Public Education, University of Washington. 

National Charter School Research Project & Center on Reinventing Public Education. (2007). Inside charter schools: A 
systematic look at our nation’s charter schools. Seattle: University of Washington. 

Nelson, H., Rosenberg, B. & Van Meter, N. (2004). Charter school achievement on the 2003 National Assessment of 
Educational Progress. Washington, DC: American Federation of Teachers. 

Peterson, P. E. (2007). School reform in Philadelphia: A comparison of student achievement at privately-managed schools 
with student achievement in other district schools. Cambridge, MA: Harvard University Program on Educational Policy 
and Governance. 

Rapp, K. E., & Eckes, S. E. (2007). Dispelling the myth of “white flight”: An examination of minority enrollment in 
charter schools. Educational Policy, 21(4), 615-661. 

Rhim, L. (1998). Franchising public education: An analysis of charter schools and private education management 
companies. Paper presented at the American Education Research Association, San Diego, California. 

Sass, T. R. (2005). Charter schools and student achievement in Florida (Working paper). Tallahassee: Florida State 
University. 

Schneider, M., & Buckley, J. (2002).What do parents want from schools: Evidence from the Internet, 24(2), 133-144. 

Schorr, J. (2000, June 5). Giving charter schools a chance. The Nation, 270, 19-23. 

Scott, J. (2008). Managers of choice: Race, gender, and the political ideology of the “new” urban school leadership. In 
W. Feinberg & C. Lubienski (Eds.),School choice policies and outcomes: Philosophical and empirical perspectives 
on limits to choice in liberal democracies (pp. 149-176). Albany: SUNY Press. 

Scott, J., & Barber, M. (2002). Charter school legislation in California, Arizona and Michigan: An alternative framework 
for policy analysis (No. Occasional Paper 40). New York: The National Center for the Study of Privatization in 
Education, Teachers College, Columbia University. 

Scott, J., & DiMartino, C. C. (2008). Hybridized, franchised, duplicated, and replicated: Charter schools and management 
organizations [Electronic version]. Forum on the Future of Public Education. http://theforum.ed.uiuc.edu/choice 
Scott, J., & Holme, J. J. (2002). Public schools, private resources: The role of social networks in California charter 
school reform. In A. S. Wells (Ed.), Where charter school policy fails: The problems of accountability and equity (pp. 

102-128). New York: Teachers College Press. 

Smith, S. (2001). The democratic potential of charter schools (Vol. 136). New York: Peter Lang. 

Solmon, L., Park, K., & Garcia, D. (2001). Does charter school attendance improve test scores? The Arizona results. 
Phoenix, AZ: Goldwater Institute Center for Market Based Education. 

The Charter School Achievement Consensus Panel. (2006). Key issues in studying charter schools and achievement: A 
review and suggestions for national guidelines (No. NCSRP White Paper Series, No. 2). Seattle, WA: The National 
Charter School Research Project, The Center on Reinventing Public Education. 

Thernstrom, A., & Thernstrom, S. (2003). No excuses: Closing the racial achievement gap in learning. New York: Simon 
and Schuster. 

UCLA Charter School Study. (1998). Beyond the rhetoric of charter school reform: A study of ten California school 
districts. Los Angeles: UCLA. 


SCHOOL CONTEXT AND CHARTER SCHOOL ACHIEVEMENT 243 


United Teachers of New Orleans and the American Federation of Teachers. (2007). Reading, writing, and reality check: 
An early assessment of student achievement in Post-Katrina New Orleans. New Orleans: Author. 

Urietta, L. (2006). Community identity discourse and the heritage academy: Colorblind educational policy and white 
supremacy. International Journal of Qualitative Studies in Education, 19(4), 455-476. 

U.S. Department of Education. (2004). Successful charter schools. Washington, DC: Office of Innovation and Improve- 
ment. 

Weiher, G. T., & Tedin, K. L. (2002). Does choice lead to racially distinctive schools? Charter schools and household 
preferences. Journal of Policy Analysis and Management, 21(1), 79-92. 

Wells, A. S. (Ed.). (2002). Where charter school policy fails: The problems of accountability and equity. New York: 
Teachers College Press. 

Wells, A. S., Grutzik, C., Carnochan, S., Slayton, J., & Vasudeva, A. (1999). Underlying policy assumptions of charter 
school reform: the multiple meanings of a movement. Teachers College Record, 1 00(3), 513-535. 

Wells, A. S., Holme, J. J., Lopez, A., & Cooper, C. W. (2000). Charter schools and racial and social class segregation: 
Yet another sorting machine? In R. D. Kahlenberg (Ed.), A notion at risk: Preserving public education as an engine 
for social mobility (pp. 169-222). New York: The Century Foundation Press. 

Wells, A. S., Holme, J. J., Revilla, A. J., & Atanda, A. K. (2005). How society failed school desegregation policy: Looking 
past the schools to understand them. Review of Research in Education, 28(1), 47-99. 

Wells, A. S., Lopez, A., Scott, J., & Jellison-Holme, J. (1999). Charter schools as postmodern paradox: rethinking 
stratification in an age of deregulated school choice. Harvard Educational Review, 69(2), 172-204. 

Welner, K., & Howe, K. (2005). Steering toward separation: The policy and legal implications of “counseling” special 
education students away from choice schools. In J. Scott (Ed.), School choice and diversity: What the evidence says 
(pp. 93-111). New York: Teachers College Press. 

Wexler, E., & Huerta, L. A. (2000). An empowering spirit is not enough: A Latino charter school struggles over leadership. 
In B. Fuller (Ed.), Inside charter schools: The paradox of radical decentralization (pp. 98-123). Cambridge, MA: 
Harvard University Press. 

Wohlstetter, P., & Griffin, N. (1998). Creating and sustaining learning communities: Early lessons from charter schools 
(No. OP-O3, CPRE Occasional Paper Series). Philadelphia: Consortium for Policy Research in Education, University 
of Pennsylvania Graduate School of Education. 

Wohlstetter, P., Malloy, C. L., Smith, J., & Hentschke, G. (2004). Incentives for charter schools: Building school capacity 
through cross-sectoral alliances. Educational Administration Quarterly, 40(3), 321-365. 

Yancey, P. (2000). Parents founding charter schools: Dilemmas of empowerment and decentralization (Vol. 135). New 
York: Peter Lang. 

Zimmer, R., & Buddin, R. (2005a). Charter school performance in urban districts: Are they closing the achievement 
gap? (RAND Education Working Paper). Santa Monica: RAND. 

Zimmer, R., & Buddin, R. (2005b). Getting inside the black box: Examining how the operation of charter schools affect 
performance. Santa Monica: RAND. 

Zimmer, R., Buddin, R., Chau, D., Faley, G., Gill, B., Guarino, C., et al. (2003). Charter school operations and 
performance: Evidence from California. Santa Monica, CA: RAND. 


Copyright © Taylor & Francis Group, LLC 
ISSN: 0161-956X print / 1532-7930 online 
DOI: 10.1080/016195609028 10179 


PEABODY JOURNAL OF EDUCATION, 84: 244-261, 2009 : 2 Routledge 


Taylor & Francis Group 


Institutional v. Technical Environments: Reconciling the 
Goals of Decentralization in an Evolving Charter School 
Organization 


Luis A. Huerta 


Teachers College — Columbia University 


This article analyzes how macrolevel institutional forces persist and limit the expansion of decen- 
tralized schools that attempt to challenge normative definitions and practices of traditional school 
organizations. Using qualitative case study methodology, the analysis focuses on how one decen- 
tralized charter school navigated and reconciled its internal objectives linked to meeting the needs 
of a Latino, inner-city student population, with the institutional pressures and demands external 
to the organization. The findings suggest that decentralized school choice options which promise 
community-based participation and control are constrained by institutionalized rule-based demands 
that limit the expansion of alternative schooling models which may hold potential for promoting 
access, voice and equity for parents, students, and teachers. 


INTRODUCTION 


The judicial and policy limitations on public school choice options in a post-desegregation context 
(Parents Involved in Community Schools v. Seattle School District and Meredith v. Jefferson 
County Board of Education, 2007) coupled with the recent Simmons-Harris v. Zelman (2002) 
decision, have set the stage for the expansion of market-based reforms that promote school 
choice alternatives in the form of private schools or decentralized charter schools. The promotion 
of schooling markets advocates for the devolution of public authority to empower local actors in 
defining and meeting the unique needs of students in their community and to create innovative 
school organizational models. Yet, as this research illustrates, when communities and school actors 
attempt to capitalize on the freedoms offered to them via the decentralized charter school reform, 
they must still uphold traditional institutional norms of schooling that may challenge and stifle 
their efforts to create new school models. As schooling markets expand in a postdesegregation 
context, it is important to understand how institutional constraints may limit alternative schooling 
models that hold potential for both promoting access, voice and equity for parents, students, and 
teachers, as well as influencing substantive change within the wider institutional environment of 
public schooling. 
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This study examines the organizational model of one northern California charter school and 
aims to discover both how the actors within the school interpreted their decentralized context and 
the challenges the school encountered as it evolved within the wider institutional environment of 
public schools. Specifically, the research focuses on whether the decentralized context in which 
the charter school operates influences the emergence of innovations, including the creation of 
new organizational models and teaching and learning strategies that challenge institutionalized 
definitions of schooling. A principle focus is on whether these innovations can, over time, resist 
the isomorphic! pressures to adopt the institutionalized norms and rule-based requirements of 
the wider institutional environment (e.g., teacher certification, instructional time, class size, 
curriculum, instruction, resource allotments, organizational structure; J. W. Meyer & Rowan, 
1977, 1978; Powell & DiMaggio, 1991). Although the promises advanced by decentralized 
school policies are a direct challenge to the existing institutional norms shared by public schools, 
we know little about how or whether the movement’s popular and political support leads to 
substantive and sustainable changes. Using institutional theory, this study aims to account for 
these unknowns by explaining and predicting both the challenges new charter organizations 
encounter and the prospects for lasting change in the broader public school institutional context. 

Presumably, the decentralized and autonomous nature of charter schools allows them to operate 
free from rule-based bureaucratic controls to challenge and redefine institutional classifications 
and definitions. Yet ironically, the challenge for charter schools to create new definitions of school- 
ing may be insurmountable because the very classifications that have resulted from bureaucratic 
controls are those that define effective and legitimate schooling. 


Research Questions and Methods 


This study focuses on the internal workings of Amigos Charter Academy (hereafter Amigos), 
a middle school serving students in Grades 6 to 8 in a traditional classroom setting in a large 
urban district in Northern California. Amigos was founded by parents and local teachers out of 
a backlash against the large junior high schools. Parents had grown increasingly frustrated with 
district administrators whom they saw as unresponsive to the needs of their children. They also 
held fears of having to send their children to unsafe schools riddled with gangs and drugs. Amigos 
enrolls 170 Latino students, nearly all of Mexican or Latin American decent. The students come 
from lower-working-class families, and more than 90% of the students are eligible for the free 
or reduced-price lunch program. Most of the students are classified as Limited English Proficient 
and many are at least 3 years below grade level on measures of academic performance. When this 





'Tsomorphism is an important concept that addresses how organizations that are responsive to an institutional en- 
vironment evolve and interact within an environment that defines or prescribes the normatively accepted practices of 
organizations. The institutional view of isomorphism is developed by Powell and DiMaggio (1991), who identified the 
following three mechanisms of institutional isomorphic change: (a) Coercive isomorphism results when organizations 
are persuaded, forced, or coerced to change by other organizations on which they are dependent. Coercive isomorphism 
is most common in organizations that operate in legal and political environments. (b) Mimetic isomorphism is linked to 
organizational uncertainty or ambiguity, which results in the imitation or mimicking of legitimate organizational struc- 
tures in the institutional environment. (c) Normative isomorphism is based on the norms of professionalization where 
organizations adapt to, or reflect the increased specialization or professionalization in a field to signal legitimacy. 

2Pseudonyms are used for the individuals, institutions, and locations named throughout this article. 
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study began in fall of 1996, the school staff comprised ten teachers, one principal, one counselor, 
and one part-time secretary and resource specialist. By the fourth year of this study in 2000, more 
than 60% of the staff had turned over and the fourth principal in as many years was leading the 
school. 

The following empirical questions drive this research: To what degree does the devolution 
of public authority to local-level actors under a decentralized school reform model result in the 
emergence of new organizational structures and new teaching and learning strategies? How can a 
charter school operating under a decentralized policy context sustain innovation within the wider 
institutional environment and gain the necessary legitimacy to ensure organizational survival? 

The qualitative case study relied primarily on observations and 35 semistructured interviews of 
various actors, including parents, students, administrators, and policymakers. Intensive fieldwork 
captured how a decentralized school policy was manifesting into practice in the context of a charter 
school organization that has a unique sociocultural ethos. Many of the findings in this single school 
case study may not be generalizable because they are dependent on the highly contextualized local 
and state policy conditions that govern charter schools in California. However, the shared policy 
context that privileges a decentralized theory of action, as well as the challenges all charters and 
other alternative forms of schooling encounter while evolving within the highly institutionalized 
environment of public schools, may make findings more generalizable across varied local and 
state policy contexts. 

The study first presents a theoretical framework derived from institutional theory that helps 
explain the environmental conditions that may challenge charter schools in their attempts to sustain 
innovation within a wider institutional environment. The framework acknowledges developments 
in new institutional theory that call for analysis of the changing educational environment through 
the lens of new institutional theory (Boyd & Crowson, 2002; H. D. Meyer & Rowan, 2006; Rowan, 
2006).? Next, the case study uses the institutional framework to examine how the organizational 
structure of Amigos was shaped both by internal human-scale organizational goals and external 
institutional norms. The discussion focuses on how, over time, charter schools may be forced to 
become isomorphic with the wider institutional environment, adopting traditional rituals—albeit 
at some levels symbolically—to garner legitimacy and “look modern.” The last section presents 
the implications of the findings and discusses how microlevel impulses of local-level actors may 
reveal more about the process of institutionalization (Zucker, 1991). 


Research on the Evolving Charter School Movement 


Charter schools have evolved as a direct response to both the bureaucratic controls that govern 
public schools in the United States and to an increasing demand for greater community control 


3Rowan (2006) explained how both the new institutionalism in organizational sociology and the American education 
system have evolved considerably since the 1970s. Thus, the changing context demands a revisit of how we might 
apply new institutional constructs that can account for both an evolving theory and its application to changing policy 
contexts, including the “tightening control of government over the core technology of schooling” linked to standards 
and accountability reforms (H. D. Meyer & Rowan, 2006, p. 8); the growth of market-based reforms, which “now seem 
central to education reform” (Rowan, 2006); and the more pluralistic interests that have led to changing power structures 
in education, including community control, that may lead to more diffuse and decentralized forms of schooling (Bidwell, 
2006; H. D. Meyer, 2006; Rowan, 2006). 


INSTITUTIONAL V. TECHNICAL ENVIRONMENTS 247 


of schools. As organizations operating in a decentralized context, free from most state and 
local regulations that govern traditional schools, charter schools represent the antithesis of the 
bureaucratic structures that govern public schools. Proponents claim that by lifting bureaucratic 
controls over schools, charter schools will act as laboratories for innovation, leading to the 
creation of local accountability systems, new teaching and learning methodologies, and efficient 
competitive school models (Finn, Manno, & Vanourek, 2000; Nathan, 1996). 

Charter school research, however, is not conclusive in determining the extent to which the 
day-to-day experiences of charters support the assumptions inherent in the decentralized theory 
of action. Only recently have substantive research efforts begun to tackle the important empirical 
issues that charter schools are raising, including charter school effectiveness and achievement 
gains realized by students in comparison to traditional school students (Carnoy, Jacobsen, Mishel, 
& Rothstein, 2005; Hoxby, 2004; S. T. Lubienski & Lubienski, 2006; Miron, Nelson, & Risley, 
2002); the creation of a competitive education market and the “tipple effects” felt by traditional 
public and private schools (Arsen, Plank, & Sykes, 1999; Henig & McDonald, 2002; Teske, 
Schneider, Buckley, & Clark, 2001); the role of parents and community in developing and 
sustaining charter schools (Fuller, 2000; Rofes & Stulberg, 2004; Schneider & Buckley, 2003; 
Yancey, 2000); creating and sustaining innovations in market-driven school reform (C. Lubienski, 
2003). 

These and other research efforts have begun to use empirical evidence and careful analysis to 
provide more critical perspectives on issues charter schools are raising. Yet the body of research 
on charter schools is still missing work that examines how local actors interpret their roles in 
decentralized school organizations, whether new innovations can be sustained or will influence 
long-term changes in the technical core (classroom instruction and student learning), or how 
innovations created in a decentralized context can support long-term institutional change. 


THEORETICAL FRAMEWORK 
The Institutional and Technical Environments of Educational Organizations 


Looking at charter school policy through the lens of institutional theory allows us to assess whether 
decentralization might influence institutional shifts that could accommodate new innovations 
created by the charter school movement, or whether the existing norms, rules, and expectations 
of the institutional environment lead decentralized schools to adopt the accepted normative 
definitions of effectiveness that are linked to bureaucratic and rule-based compliance. 

An institutional theory perspective assumes that charter schools, like all public schools, are still 
operating within a highly institutionalized environment. The institutional environment identifies 
legitimate forms of schooling via a school’s conformity to environmental conditions that define 
effectiveness through inputs like standardization and certification procedures (the symbolic rituals, 
norms, and myths of the institutional environment) rather than measurable outcomes (Meyer & 
Rowan, 1978). By concentrating on coordinating ritual classifications rather than the technical 
aspects of education, schools can avoid controlling their outputs and avoid inspection that can 
call attention to the lack of coordination between instruction and outputs. 

To challenge successfully the existing and robust institutionalized definitions of effective 
schooling, decentralized school reform policies will have to make si gnificant inroads in redefining 
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normative criteria of effective schools, both at the local level and in the wider environment. How- 
ever, institutional theory predicts that “organizations fail when they deviate from the prescriptions 
of institutionalizing myths: quite apart from technical efficiency, organizations which innovate 
in important structural ways bear considerable costs in legitimacy” (Meyer & Rowan, 1977, 
p. 353). In other words, organizations that challenge institutionalized conceptions of effective 
schooling may be seen as ineffective and risk their legitimacy. The challenge for charter schools 
will be to prove the legitimacy of their innovations independent of institutionalized definitions of 
effectiveness (Huerta, 2002; Huerta & Zuckerman, 2007). 

As charter schools seek to distance themselves from traditional schools—rejecting the bureau- 
cratic controls that advocates claim have stifled effective schooling and innovation—they begin 
to adopt efficiency criteria and organize around their technical core rather than the institutional 
environment. Powell and DiMaggio (1991) explain how an organization that adopts efficiency 
criteria and rejects conformity to institutionalized rules, undermines “ceremonial conformity and 
sacrifices its support and legitimacy” (p. 41, see also J. W. Meyer, Scott, & Deal, 1983). By 
attempting to tightly coordinate the internal workings with the ritual classifications, schools risk 
devaluing the meaning of ritual classifications which have bought schools public legitimacy (J. 
W. Meyer et al., 1983; Powell & DiMaggio, 1991). Whether charter schools can sustain organi- 
zational structures that are responsive to their technical environment while still operating within 
the wider institutional environment is the principle theme that drives this analysis. 

The case study that follows illustrates how Amigos attempted to advance its own definitions 
of effectiveness independent of bureaucratic rules. Amigos adopted technical efficiency criteria 
in attempts to distance itself from the demands of the institutional environment. In doing so, it 
organized around the demands of its technical environment, attempting to link its organizational 
structure explicitly to the activities of its technical core. However, the challenge for Amigos 
was to operate effectively within an environment that did not buffer the technical core through a 
formal structure designed to respond to traditional bureaucratic efficiency demands. Turning to its 
technical environment proved to be the school’s biggest challenge as it was further challenged to 
gain legitimacy within the wider institutional environment. This resulted in increased ambiguity 
within its organizational structure. 


CASE STUDY 
Testing the Promises of Decentralization at Amigos Charter School 


This case study focuses on how Amigos Charter Academy evolved over its first 6 years of 
existence, from its inception in 1993, to its charter renewal bid in 1998, to | year after being 
granted a renewed charter in early 2000 (for an expanded case study, see Wexler & Huerta, 2000). 
This section addresses the first research question and explains how actors were empowered to 
create a small human-scale collective of teachers, parents, and community members and were 
committed to meeting the unique needs of their students without bureaucratic directives. The 
remaining section addresses the second research question and describes how in its approaching 
charter renewal bid, Amigos was forced to shed its renegade ideals and cast for lifelines in the 
external environment to gain legitimacy and insure organizational survival. 
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Innovations at Amigos Charter Academy 


Amigos was successful in implementing new innovations in both its organizational structure 
and its teaching and learning methodologies. Amigos founders wanted to create a school that 
revolved around the values of the community that it served. Parents of Amigos’ students and 
their community had continually seen their demands to the local school district go unmet. A 
founding parent described the experience: “When we went to the school district for meetings 
(board meetings), sometimes we would stay there until midnight, waiting for someone to listen to 
us. But they didn’t want to listen to us.” Frustrated with remote and seemingly indifferent school 
officials, the original founding parents and teachers built Amigos upon a shared commitment to 
parent and community involvement. According to the Amigos charter: 


Empowerment of staff, students, and parents will be a visible reality. Active and meaningful partic- 
ipation of parents and students, along with leadership roles taken by all staff, will result in a school 
where everyone has a voice and a contribution to make in the development of this middle school. 


The founders believed that if parents and their immediate community were empowered with the 
ability to create their own school, the needs of both students and parents could be better met. 

Amigos staff and parents worked hard to implement a curriculum that met the needs of its 
Latino students, many of which were Spanish-only speaking or limited-English proficient. A 
direct result of frustration with the district, the creation of a “newcomers” curriculum was one 
of the principle objectives of the school’s founders. Amigos enjoyed success in its newcomers 
curriculum, which served newly arrived immigrant students with specific language and social 
adjustment needs—the most sensitive sector of its population. Students who had recently arrived 
to the United States were not mainstreamed into the general school population but rather were 
given intensive ESL classes for half their instructional day and then were allowed to participate in 
mainstreamed classes, usually math and science. The newcomers class proved to be a success and 
was an innovative alternative to what Amigos teachers characterized as the “Mickey Mouse ESL 
classes.” which immigrant students were tracked into at more traditional schools. At Amigos, 
these students did not “fall through the cracks,” but rather they were nurtured and provided with 
the tools to begin the difficult transition into mainstream classes. 

Mainstream students were taught in small classes which contrasted to the overcrowded class- 
rooms of the other larger middle schools in the district. Teachers also experimented in creating 
student-centered curriculum that did not rely on standardized lessons derived from work books 
and textbooks. Yet over time, the challenge in creating new curriculum with limited resources 
and support proved taxing for teachers. 

To ensure that the school organization was structured to meet the needs of democratic local- 
ism, Amigos also attempted to implement a school-based management governance structure that 
involved both parents and community members in the school’s decision-making process. Found- 
ing teachers believed that flattening the hierarchy would allow equal voice to all stakeholders. 
They acknowledged that to help the population of students they were serving—students whose 
needs would have been shuffled-away by the unresponsive bureaucracy of the district’s middle 
schools—they would have to create a school environment that provided fewer access barriers to 
the school community. Sustaining the governance structure proved to be very difficult and led 
to many struggles that challenged the school actors and their ability to sustain a viable school 
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organization. However, through its shared governance ideals, the school did succeed in creating 
a rich human-scale organization that empowered parents, teachers and students in a variety of 
capacities that were nonexistent in the local traditional schools. 


Sustaining Local-Level Change and Innovations 


The local conditions that sustained innovations at Amigos are important components of the story 
and critical to understanding how local-level conditions are challenged by the wider institutional 
environment. At Amigos, these conditions included not only the high degree of autonomy that 
the school fought hard for and struggled to retain at all costs but also the will of the actors who 
participated in the rich democratic school community. Even before turning to the day-to-day 
challenges of creating a viable school organization, the school actors worked hard to build a safe 
and familiar community with students and parents. 

Even though Amigos struggled to implement an innovative organizational structure, the small 
successes amounted to changes that may not have been possible in the traditional bureaucratic con- 
text. The nature of its decentralized context allowed the autonomy necessary to at least experiment, 
even if over time its innovations failed to become institutionalized in the school’s organizational 
structure. Whether institutionalized or not, the innovations that Amigos experimented with were 
inroads to changing a stagnant inner-city school district. Amigos had successfully managed to 
promote a small school size, small class sizes, a safe and nurturing school environment, and a 
group of teachers with personal commitments to the school’s educational mission. In light of 
these innovations, the challenge for the organization came when the school was faced with the 
substantive teaching, learning, and organizational issues that needed to be created if the school 
was going to succeed in effectively educating its students. 

The next section describes how Amigos strived to maintain and advance innovations when it 
began to adopt technical efficiency criteria and tightly link its instructional and accountability 
models to the technical core—a direct challenge to the rule-based accountability models advanced 
by the institutional environment. 


Breaking Away From the Bureaucracy 


In many ways Amigos represents a renegade school attempting to escape from the state’s hege- 
monic stronghold over standardized schooling models. Its founders struggled to purge their new 
organization from any state or district influence, which could impose on the day-to-day activities 
of the school. The founding principal explained, “The less we have to do with them [the district] 
the better. ... What the hell, we don’t need them.” Her sentiment was echoed by one of the 
founding teachers, who explained the rights bestowed on them by their autonomy: 


Because we are autonomous we could do things differently. . . there is the opportunity to pretty much 
do anything you want. There is a certain advantage in hiring who you want to hire. We can be what 
we want to be, choose the site that we want, we can develop our own standards, we can create our own 
academic program, we can develop our own bilingual programs, we are not constricted by unions’ 
guidelines or other district rules. 
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Their intent was to keep the state out of the school at all costs. For example, the school 
secured its own private student liability insurance policy, rather then staying within the district’s 
insurance pool. The decision came at a significant cost to the school, amounting to more than 
$10,000 annually, compared with only $400 for the district-sponsored insurance. The fiscal burden 
was written off as the cost of remaining autonomous and reflected Amigos’ steadfast belief that 
foregoing relationships with district officials would ensure the school’s long-term autonomy. 

Initially, Amigos did not make explicit claims of advancing efficiency, at least not with respect 
to its organizational model; however, the school did make claims about its ability to meet the needs 
of its student population by virtue of its decentralized freedoms. Determined to exercise the limits 
of its new freedoms, Amigos rejected institutionalized definitions of effectiveness in exchange for 
locally created definitions that were more consistent with what it felt were the unique needs of its 
students. Amigos consciously sought to shut itself off from the prescriptive ritual classifications 
of the institutional environment. A young teacher in her first year teaching at Amigos explained 
how creating new curriculum to fit the unique needs of her students was a challenge: “There is an 
overriding tone among staff and administrators that if you need a traditional text book to teach 
from then you are a bad teacher.” As the school distanced itself from the local district and broke 
ties with the bureaucratic demands that were seen as burdensome elements of ineffectiveness, it 
turned inward to its technical environment and chose to operate as a closed system. By rejecting 
institutionalized definitions, rules, and rituals, it was left only with the job of coordinating its 
internal structure. What resulted was a school organization that struggled to effectively organize 
around the activities of its technical core. 


Organizing Around the Technical Core 


By turning inward and operating as a closed system, Amigos decided to shift from ceremonial 
conformity to focus instead on the “concrete effects” of its technical activities (J. W. Meyer & 
Rowan, 1977, p. 355). Amigos attempted explicitly to link its organizational structure that was 
grounded in local accountability, with the technical activities of the school to make the day-to-day 
operations more transparent. One teacher explained: 


There is a clean-slate effect here. There is so much to do as opposed to undo. There is not a lot of 
bureaucracy you have got to work around. There are not a lot of systems you have to learn to deal 
with. But we have to develop our own policies and simultaneously make them work .. . everything 
from curriculum to school uniforms to detention policies. This is really an open slate. There are only 
students who need teaching to and parents with a lot of confidence that this is a better school and 
environment than they could otherwise be sending their kids to. 


However, because Amigos actively challenged the legitimacy of what it viewed as the insti- 
tutionalized rituals and myths of the wider environment—choosing not to advance a logic of 
confidence sustained by ceremonial conformity or create a formal structure that would buffer 
the inner workings of the organization—the school’s internal structure was left vulnerable. As a 
closed system, the school had to prove its efficacy by advancing technical efficiency measures 
linked directly to its technical core. 
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The school also experimented with a governance structure built upon the democratic principles 
of shared management. In efforts to implement a school-based management approach, the school 
staff delegated traditional administrative duties to teachers, and shared other governance duties 
with parents and community members. Yet attempts to coordinate the tasks and purposes of the 
technical core led to countless disagreements among teachers and administrators. Endless staff 
meetings resulted in teachers feeling both defeated by the amount of work they were charged to 
do, and confused about how to formally define their roles. One teacher lamented how “I would 
honestly say that this year I have probably spent 80 percent of my time on administrative duties 
and only 20 percent on teaching.” Another teacher explained, 


The process of shared decision making may be democratic, but it tends to be over-powered by 
those who have been here longer ...they voice their opinions louder and are often heard. Also, 
collaboration is lacking in general and .. . it is more individualistic than anything else. Everyone is 
really overwhelmed and just trying to survive. At every meeting we seem to be dealing with how the 
school is being run and less time developing curriculum, which is my biggest concern. In fact, we 
have only concentrated on curriculum development three times over the course of the entire year. 


This disarray was representative of the ambiguity the organization was facing in attempting to 
rationally link its technical work activities to its organizational objectives. 

Recall that organizations that engage in advancing a logic of confidence, and conform to 
ritual classifications of the institutional environment, will create a formal structure that buffers 
the layers of the organization and shields the technical core from uncertainties in the outside 
environment (J. W. Meyer & Rowan, 1978; J. W. Meyer et al., 1983). In the context of Amigos, 
the school was unsuccessful in creating an effective formal structure because it was unable to 
reconcile its technical activities with its organizational goals. Without the traditional institution- 
alized means to define its work activities and buffer the technical core, the school struggled in 
defining both effective teaching and learning methodologies for all students, and in creating an 
organizational structure that could advance the objectives of the small collective. The nature of 
Amigos’s small organization, its disconnect from the external environment, as well as its inter- 
nal shared governance structure, led to an organization that had few layers in its structure. In 
essence the organizational structure of Amigos was vulnerable to external scrutiny—without a 
formal structure to buffer the internal workings of the organization. There was no decoupling of 
organizational layers and there was no loyalty to advancing external institutional requirements. 

Amigos did not intentionally set out to adopt technical efficiency criteria, but rather intended to 
redefine effective teaching and learning practices for its unique student population. It succeeded 
in reaching out to certain students. However, it failed in creating an organizational structure that 
could sustain its technical activities. Because the school rejected institutionalized rules, it was now 
subject to the demands of its local technical environment. The internal structure went unbuffered, 
allowing scrutiny and inspection not only by outside actors but also by actors within the school 
organization who questioned the efficacy of many of its practices and led to disagreements and 
divisions among those who were committed to sustaining the organization. One teacher described 
meetings as “really volatile in terms of people feeling the threat of the mission, the vision being 
changed, the school being taken away from the people who had fought for it and created it.” 
Senior teachers felt their voice should carry more weight, whereas younger teachers wondered 
whether the school was truly democratic: “When I was first hired it was presented to me as ‘this is 
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our school, we’re empowered to make decisions,’ but that’s really not the case.” Another teacher 
spoke of the challenges of governing democratically and creating a shared culture. Thus, the small 
collective was left to create and sustain a tightly-coordinated, locally accountable organization 
structured around the demands of its technical activities. 

By the fourth year of operation, attempts to make its technical production explicit and clearly 
linked to the school’s organizational structure proved too taxing. Apart from never being able 
to create an organizational structure that aligned with the mission of the school, the staff was 
also unable to define criteria of effectiveness. Amigos had failed to create a viable organizational 
structure, making its technical activities subject to scrutiny—by school actors, its school board, 
and ultimately the district school board. In time, the school ultimately failed to rationally orga- 
nize its work activities, create a clear division of labor among school actors, manage specific 
organizational procedures, and create a sustainable organizational structure. 


Teachers Seek Anonymity Within Their Organization 


During its fourth year of operation with a charter renewal bid looming in the midst of the 
following academic year, the Amigos staff was still struggling to implement a sound curriculum 
and formalize an organizational structure that was effective in managing the day-to-day operations 
of the school. With hopes that the district would render Amigos another 5-year charter, the school 
began identifying the criteria which the district might use in evaluating its effectiveness. With 
90% of the students performing below grade level and 70% of them identified as limited-English 
proficient, the school was challenged in defending the effectiveness of its academic program. One 
teacher identified this difficult challenge: 


Soon we’ll have to defend the charter and we have to prove to the district that we’ve made progress 
with the kids and show them that we have created our own curriculum and standards. If we are not 
going to use test scores then we have to have something else. We want to assemble portfolios to show 
that even if the kids are not ranking highly on the tests that they have improved since they have gotten 
here. I think the most powerful evidence is testimonials from the kids and parents. But we realize this 
won't carry us. 


As the charter renewal date approached, the local school board also scrambled to begin 
identifying the protocols that would measure the school’s effectiveness. As the Amigos sponsoring 
agency and the body responsible for evaluating the school’s progress, the board was challenged as 
it had not yet lived through the district’s first charter school renewal bid. The board was uncertain 
about their level of discretion and which, if any, state or local guidelines sponsoring agencies 
were expected to follow as criteria for renewal. 

The school was at a juncture that led to self-reflection and self-doubt for Amigos’s staff. They 
quickly realized that the process of self-evaluation was difficult, considering that the majority 
of the school’s goals were yet to be achieved. At staff meetings, teachers questioned whether 
they would grant Amigos a 5-year charter renewal if they were members of the board. The 
staff reflected on how they continually failed to implement even the most simple governance 
procedures as well as a sound curriculum for all of their students. A teacher explained how: 
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In terms of established general curricula, I think there has been very little ... that whole idea of an 
integrated curriculum has sort of fallen by the wayside. The main reason why is that there is lack 
of staff continuity, lack of time to talk about those issues, and also partly because of the staffing 
problems and the size of the school. 


Considering the barriers that teachers and parents had encountered along the way, Amigos had 
already made significant positive changes that benefited their students (e.g., newcomers curricu- 
lum, class size, school size, safe school environment). However, over time, the determination to 
create a human-scale institution proved overwhelming. Amigos was still struggling with so many 
basic needs, including paying utility bills, securing and reconditioning facilities, and providing 
basic supplies to its teachers. One of the veteran teachers explained: 


I think that part of the problem that we have had here historically is that we were a start-up school. 
You know, we have to deal a lot with survival issues, its basic survival. And I think that has been what 
has sort of gotten in the way of improving and creating a school that we want to see. For example, 
last year we went without electricity until mid-November. 


Teachers at Amigos also struggled with the shared decision-making process of their school- 
based management plan, a school governance structure that would have proved challenging for 
even the most stable and cohesive school organizations. The teachers longed for the ability to 
simply teach the students they were charged to educate without worrying about their shared 
administrative duties. They also longed for the top-down administrative structure common to 
traditional school organizations, hoping that such an arrangement would buffer their teaching 
duties from the burdensome administrative tasks. A first-year teacher explained: 


Structure is lacking; there is nothing formalized. We should definitely work toward putting more 
traditional administrative and organizational structures in place! We constantly have to create things 
that we probably could have avoided doing if those structures were in place. 


At staff meetings the teachers began to speak of “how things should be” according to their own 
interpretation of traditional administrative structures. They engaged in hypothetical discussions 
of what Amigos would be like if they adopted a clear-cut line of administrative responsibilities. 
The energy teachers expended to address tasks normally performed by traditional administrators 
were taking their toll in the classroom. 

In essence, the teachers were longing for what Meyer, Rowan, Scott, Weick (J. W. Meyer 
& Rowan, 1977, 1978; Scott, 1987; Weick, 1976) and others have described as the decoupled 
organizational model of schooling. The teachers wanted a “new” division of labor in their school 
organization, complete with a hierarchical formal structure that would buffer the inner workings 
of their classrooms from administrative demands. They found that juggling both administrative 
and teaching duties proved overwhelming. Even though the teachers believed and understood the 
objectives of their governance program, they wished for the convenience of assigning adminis- 
trative duties to an administrator and teaching duties to the teachers. In addition, attempting to 
maintain the school’s organizational structure closely linked to the technical core was a daunting 
challenge which the teachers were tired of, and was quickly leading to burnout. 
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In a sense, the teachers sought anonymity and independence within their own human-scale 
collective that they had struggled to create. They realized that after several years of addressing the 
same administrative issues in weekly staff meetings, their organization was beginning to “come 
apart at the seams.” However, they wavered as they struggled to keep the organization afloat. 
They began to question whether adopting a hierarchical administrative structure would reduce 
the participatory nature of the school organization—a goal that was paramount to the original 
mission of the school. They also asked themselves whether adopting a traditional governance 
structure would result in looking like any other district school. These ongoing discussions and 
events marked a turning point in the school’s organizational evolution. 

Hypothetical discussions soon turned into action. Feeling deserted in their failed attempts 
to sustain their organization, teachers and administrators alike began to explore the fiscal and 
organizational safety net that district services would provide under a more traditional school 
model: from the most basic necessities, such as food services, janitorial services, transportation, 
liability insurance, and accounting services, to more substantive needs, including curriculum, 
supplies, text books, library books, professional development, and adequate facilities. Late in 
the school’s fourth year, Robin—the third principal in the school’s short history—defied the 
warnings of the founding principal and began to forge external relationships. She started by 
soliciting the help of influential community groups—beyond those that had supported the school 
in its infancy—to aid the school with resources including both human and social capital. Reaching 
out for lifelines in the community eventually yielded several new members for the Amigos School 
Board, but it also resulted in “squeezing-out’” parents of Amigos students; only two remained on 
the seven member board. The inclusion of individuals external to the school brought forth what 
the founding mother of the school had warned against—the school would now have to sacrifice 
its own objectives to the interests of outside actors. 

As the school began to forge new external relationships, the outside actors began to inquire 
about the school’s practices. At Amigos School Board meetings they asked, Why were there no text 
books at Amigos? Why was there no cafeteria? Why were the facilities so dank and dilapidated? 
Why did teachers share restrooms with students? Why were all teachers not certificated? Why 
were students not taking the standardized test? Why was there no formal curriculum or academic 
standards? Why were teachers sharing administrative duties with the principal? These important 
questions led to a redefinition of the school’s objectives and organizational structure at the expense 
of the school’s original intent to deflect any outside influence, which could derail the mission of 
the school. 


Succumbing to lsomorphic Pressures From the Institutional Environment 


As Amigos prepared for its charter renewal bid early in its fifth year of operation, the school 
began adopting a more traditional governance structure. The distinction between teacher and 
administrator became more evident. Teachers were responsible for their teaching and learning 
duties and the lead teacher, school counselor, and principal were responsible for the administrative 
duties. The tone at staff meetings quickly changed. Teachers no longer spent countless meetings 
teasing out the logistics of sharing administrative duties. Job titles were clear, and staff were 
responsible for the duties of their respective titles: teacher, counselor, administrator. In a few 
months the adoption of a more traditional governance structure led to better coordination. As 
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one teacher explained, “It was easier to finally get some of our work done by ourselves, without 
having to share and explain everything to everyone at staff meetings.” 

The school and its actors then turned their attention to how their locally conceived definitions 
of effectiveness were not aligned with what the school board would expect from them during 
the process of evaluation and renewal of the school’s charter. Robin was informed by a district 
assistant superintendent what was expected of Amigos in preparation for the charter renewal 
bid. Robin learned that the district would hold Amigos to institutionalized definitions of effec- 
tiveness that amounted to whether Amigos looked like a traditional school (e.g., staffing ratios, 
standardized academic programs, certification of teachers, hierarchical governance, measurable 
student outcomes on standardized tests, attendance and dropout rates). The harsh news meant that 
several of the original school goals that Amigos had succeeded in implementing—a small and 
safe school environment, increased parent participation, small class sizes, meeting the needs of 
limited English students—were not measurable by institutionalized definitions of effectiveness. 
The directive from the district was not punitive, it simply reflected the district’s ambiguous role in 
overseeing Amigos, its first charter school. The school board had not prepared alternative criteria 
that could account for Amigos’s progress on its locally defined goals. Instead, it would evaluate 
Amigos using institutionalized norms of accountability. 

The renewal process quickly became an exercise in survival, which left Amigos no choice but 
to surrender to the isomorphic pressures of the wider institutional environment. The school began 
to adopt traditional definitions of effectiveness, including the creation of a formal structure, as 
well as adopting teaching and learning methodologies consistent with institutionalized definitions 
of schooling. 

In the months preceding the renewal bid, Robin began the next steps toward forging a stronger 
relationship between Amigos and the district but at the expense of further alienating herself from 
the founding teachers and parents. Amigos began formalizing its relationship by tapping the 
district for fiscal and material resources, which it had gone without during its first 4 years of 
operation. For example, Robin began the process by urging district officials to place Amigos’ 
students back on the district roster to assure that it would receive not only both state and 
federal funding that was due to them but also some simple services that students in other district 
schools were receiving. Ever since the school’s inception, the district had dropped all Amigos 
students from its roster. The lack of proper record keeping by the district—especially vital student 
demographic information—amounted to a shortfall in certain categorical funding programs. Most 
important, Amigos was unable to collect its fair share of federal Title I (poverty) and Title VII 
(bilingual education) funds because socioeconomic status and other demographic information of 
Amigos’s students were no longer collected by the district. Similar funds from state categorical 
programs also went unclaimed or unsolicited. The extra funds would have proven extremely 
helpful had they previously been granted to Amigos. District officials would now keep track of 
students who entered Amigos on central files, determine the amount of categorical funds Amigos 
should receive from the state and federal governments, and pay for a resource specialist to comply 
with special education codes. 

Robin then turned to soliciting textbooks, curriculum materials, and standardized testing 
materials from district officials. Robin, along with the teachers, had grown anxious with not 
being able to serve students with proper instructional materials. Although teachers were com- 
mitted to delivering a student-centered curriculum, their original goal to use nonmainstream 
and teacher-created materials was overrun by the pragmatic realities of sustaining their school 
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organization. Robin also persistently negotiated for library funds and more regular visits by 
the district book mobile. She also requested that Amigos’s teachers be included in the district- 
sponsored professional development programs. Last, she requested that Amigos be allowed to 
return to the district’s student liability insurance pool. This saved the school nearly $10,000 
annually. 

Robin also reinforced the school’s relationship with the Coalition of Essential Schools, the 
influential school reform collaborative that had provided the school technical assistance over the 
previous 2 years. A strong partnership with the Coalition was vital at this stage and Robin made 
certain to exploit the organization’s services. The Coalition began acting as a liaison between the 
district and Amigos, playing an important role in negotiating the criteria that would be used to 
evaluate Amigos. At the school level the Coalition continued providing technical assistance and 
helped Amigos implement national education standards in math and science. 


Symbolic Institution-Building Earns Amigos a New Lifeline 


By the time Amigos reached its renewal bid in May 1998, the school had forged important 
relationships with the school district, and partnerships with external organizations. The new 
bonds rendered the school a degree of legitimacy that was absent during the school’s explicit 
renegade period of years past. Amigos was now prepared to demonstrate to the school board that 
it had adopted textbooks, curriculum, testing materials, and academic standards sanctioned by the 
district. In essence, the school was now upholding both institutionalized definitions and popular 
expectations of what a school should look like. From the perspective of institutional theory, J. 
W. Meyer et al. (1983) describes how “schools succeed and fail according to their conformity 
to institutional rules, rather then by the effectiveness of their technical performance” (p. 56). 
Amigos succumbed to the isomorphic pressures of the institutional environment when it realized 
that its survival depended on its ability to adopt the ritual classifications that district officials 
characterized as legitimate schooling. 

The district’s coercive pressure—explicit evaluation criteria for charter renewal—forced the 
school to become isomorphic with traditional conceptions of effectiveness advanced by the insti- 
tutional environment (Powell & DiMaggio, 1991). The school actors recognized that the survival 
of the organization depended on their ability to adopt a formal structure that looked legitimate— 
even if only symbolically—in accordance with institutionalized definitions of schooling. The 
isomorphic pressure resulted in Amigos’s successful adoption of a formal structure, including 
myths and rituals consistent with the regulatory demands of the institutional environment. J. W. 
Meyer and Rowan (1977) explain that 


by designing a formal structure that adheres to the prescriptions of myths in the institutional environ- 
ment, an organization demonstrates that it is acting on collectively valued purposes in a proper and 
adequate manner. The incorporation of institutionalized elements provides an account of activities 
that protects the organizations from having its conduct questioned. The organization becomes, in a 
word, legitimate, and it uses its legitimacy to strengthen its support and secure its survival. (p. 349) 


The actions of Amigos’s staff were an important first step in establishing legitimacy and 
securing the survival of their organization. 
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The coercive isomorphic pressure to change was intensified by the dependence of Amigos on 
the district. Powell and DiMaggio (1991) suggest that an organization which depends on another 
for “funding, personnel, and legitimacy” will be more compelled to adopt the organizational 
structure, culture, and climate of that organization. Ironically, in the case of Amigos, although 
the school operated in a radically decentralized context from the district, it was still subject to 
isomorphic pressures from the district because of important resource flows on which Amigos 
depended. More important, Amigos’s legal and regulatory dependency on the district allowed the 
district to wield coercive isomorphic pressure on Amigos, eventually forcing the school to adopt 
institutionalized conceptions of effectiveness. 

The renewal process became very political as the school board was compelled to renew 
Amigos’s charter on the basis of community support alone. After several contentious meetings 
where board members were confronted by hundreds of community members and activists who 
supported Amigos, the school was granted a 5-year charter renewal by a unanimous vote and 
given a new lifeline. The school board welcomed Amigos to the district—symbolically as well 
as substantively through its new administrative agreements—and reminded Amigos that over the 
next 5 years it would be held to the academic goals that it promised. The formal organizational 
structure along with the adoption of important ritual classification held sacred by district officials, 
resulted in a new level of symbolic legitimacy for Amigos. 


Re-evolution of Amigos Under Normative Definitions of Schooling 


During one of the last visits to Amigos in early 2000, nearly a year after its charter was renewed, it 
was evident that the adoption of ritual classifications was indeed symbolic on the part of Amigos 
staff, as textbooks and other curriculum materials secured from the district were seldom used, and 
the adoption of academic standards had yet to be formalized into the curriculum. Yet, the school 
organization had successfully created a new formal structure, and advanced a logic of confidence 
to the wider environment symbolically proclaiming that the school was engaged in legitimate 
forms of schooling. Internally, its loosely coupled governance structure buffered the technical 
core from external demands and inspection from a variety of organizational layers, including the 
state, the district, the Amigos School Board, and its parents. In essence, the school had earned a 
new degree of legitimacy within its new institutional environment as well as a level of stability 
linked to its adoption of ritual classifications. 

However, the school still faced important challenges at the level of its technical core. Amigos 
was challenged again by ambiguity that was rooted in its steadfast efforts to remain decentralized 
from the district, and efforts to advance its original objectives through an organizational structure 
that was closely linked to the technical core. Influenced by the few remaining founding teachers, 
a small minority of teachers attempted to dismantle the traditional organizational structure, and 
return to the democratic and participatory human-scale organization of its past. However, the 
loyal veterans were now in the minority and were challenged by a new generation of teachers 
who did not share the same reverence for the school’s original organizational objectives. In 
the end, the school faced new challenges as it struggled to identify an organizational structure 
that could exploit its decentralized freedoms while advance attempts at symbolic institution 
building. 
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CONCLUSION 


This research is consistent with recent literature that explains how charter schools are illustrating 
that innovations in teaching and learning and school organizational models are possible as a result 
of their decentralized context (Henig & McDonald, 2002; Rofes & Stulberg, 2004; Schneider 
& Buckley, 2003). However, they are challenged by the institutionalized conditions that protect 
the status quo of the institutional environment. Even incremental changes that could influence 
institutional shifts and accommodate innovations prove to be challenging under these conditions.* 

This research provides insight into how the normative frame constructed by macroinstitu- 
tionalism provides significant explanatory and predictive value that identifies the institutional 
constraints that lead to isomorphism. In addition, the micro-organizational data that was pre- 
sented complements the macro-organizational perspectives and provides insight into the nuances 
of how organizations develop and the agency that actors within organizations possess in navigat- 
ing institutional pressures—described by Zucker (1991) as the “cognitive processes involved in 
the creation and transmission of institutions” (p. 104). 

The findings suggest that isomorphism with the wider institutional environment may earn 
charters legitimacy within the public school system and may compel charter school organizations 
to adopt many of the same structures as traditional schools. However, internal organizational dis- 
sonance linked to the challenges of operating an independent school, as well as policy pressure for 
increased regulatory demands on charters, may lead to organizational ambiguity, and ultimately 
a school reform model that is co-opted by traditional definitions of schooling. 

The Amigos case illustrates many of the challenges and constraints that all charters may face 
as they try to occupy the margins between public accountability and communal initiative and 
whether over time the promises and intents of decentralized school policies will actually reinvent 
or revolutionize school organizational models. This research also speaks to the limitations that 
even market-based choice models encounter when creating different schools that may be better 
attuned to the unique needs of diverse communities. In a postdesegregation policy context the 
expansion of state-sponsored decentralized and market-based choice programs must be weighed 
against state-centralized accountability programs to determine which reform option holds stronger 
promise in providing equity and access to quality schooling for disenfranchised communities. 
This is especially important in a context where rigid standards and accountability policies like No 
Child Left Behind may further limit the ability of states to address the needs of underperforming 
students. As charter and other market-based models of schooling expand, additional research 
must weigh how the changing discourse of legal, policy and regulatory schemes either foster or 
reject the development of new schooling models that may challenge institutionalized forms of 
schooling. 


4In addition, the findings counter the assumption that simply devolving public authority to local actors will empower 
communities to create and sustain their own schools. For many low income communities with limited social, human, and 
hard capital, the process of activating resources to affect change and insure survival of a school organization can prove 
to be a daunting task. Specifically, public authority is not a commodity that can simply be devolved to local actors and 
result in empowerment (Handler, 1996). Instead, empowerment through decentralization is a process that must account 
for local economic, political, and social/cultural conditions and the ability of local actors to activate public authority. 
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In contrast to unregulated school choice, regulated choice programs oversee the assignment of 
students to schools with equity in mind. This article puts forth evidence for three claims with respect 
to unregulated and regulated school choice: (c) Unregulated choice plans tend to exacerbate the 
stratification of students along race, class, and achievement lines; (b) regulated choice programs 
have the potential to increase the integration of schools and, at the very least, to prevent further 
social stratification; and (c) the evidence that suggests unregulated choice programs lead to improved 
academic achievement or curriculum innovation is unconvincing. By influencing the racial and social 
class composition of schools, choice programs help determine the human capital obtained by students 
and families. Exposure to new cultural and social forms of human capital can lead to enhanced life 
opportunities for all children, but particularly to those who do not otherwise have access to such 
capital in a hegemonic society. Considerations of social justice suggest that policymakers should 
continue to search for ways to design school choice programs that promote integration. 


The authors in this special issue examine school choice from a variety of disciplinary perspec- 
tives. The findings speak to the effects of choice on student outcomes, curriculum innovation, and 
the social composition of schools, and they offer insights into the school choices families make. 
The implications for education equity are considered throughout the issue. 

This closing article serves four purposes. First, it distinguishes regulated and unregulated 
forms of school choice. In contrast to unregulated choice, regulated choice programs oversee the 
assignment of students to schools with equity in mind and typically provide additional supports to 
children and families from disadvantaged backgrounds. Second, we synthesize the policy relevant 
empirical findings that can be drawn from the set of articles in this special issue and the related 
literature. In doing so, we put forth three claims: (a) Unregulated choice plans tend to exacerbate 
the stratification of students along race, class, and achievement lines; (b) more than unregulated 
choice programs, controlled choice programs appear, in some cases, to increase the integration 
of schools and, at the very least, to prevent further social stratification; and (c) the evidence that 
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suggests unregulated choice programs lead to improved academic achievement or curriculum 
innovation is weak or substantially lacking. 

The third aim of this article is to examine the policy implications of these findings. By 
influencing the racial and social class composition of schools, choice programs can help determine 
the human capital bestowed on students and families. Exposure to new cultural and social forms 
of human capital can, in turn, lead to enhanced life opportunities for all children, but particularly 
to those who do not otherwise have access to such capital. Considerations of social justice 
suggest the potential effects of choice on disadvantaged students are of paramount importance in 
designing school choice plans. Last, this article concludes with recommendations to policymakers 
and researchers of school choice. Policymakers are encouraged to continue to search for ways to 
design school choice programs that promote integration, and researchers are urged to continue 
efforts to assess the effects of specific types of choice program on school composition and student 
outcomes, and also to assess the effects of peer environments on individual students’ lives. 


THE REGULATORY ASPECTS OF SCHOOL CHOICE 


School choice comes in many different forms. Within the private sector are secular and nonsecular 
schools that can be accessed by tuition paying families or, in far fewer cases, through scholarship 
or means tested voucher programs. Within the public sphere are charter schools, magnet schools, 
and traditional public schools that can be accessed through intra- and interdistrict open enrollment 
plans, voluntary integration plans, or more regulated desegregation programs. Great variability 
exists within any one of these options. For example, a charter school in Michigan can be chartered 
by the school district, run by a profit-making education management organization, have no 
entrance requirements, and be geared toward a college preparatory curriculum. A charter school 
in Arizona may be chartered by any one of three different sanctioning agencies, run by a local 
governing council, have an audition entrance requirement, and be geared toward supporting 
students’ artistic development. 

One way to classify school choice programs is by the degree of regulatory oversight over student 
assignment and selection into schools. When a governing authority regulates student assignment, 
it is referred to as a “controlled choice” program. Controlled choice programs consider a range 
of student and school characteristics to balance school enrollments by race, family income, or 
achievement. Some controlled choice plans are the result of court orders or attempts to preempt 
legal action, and still others are conscious attempts to promote equitable peer environments. 

Controlled choice plans have changed dramatically since they were introduced following 
Brown v. Board of Education. Legally sanctioned, race-conscious controlled choice plans have 
dwindled considerably over the past 2 decades as a result of more conservative interpretations by 
courts at all levels (Orfield & Lee, 2007).! Race-conscious assignment schemes that are not part 
of court-ordered desegregation plans face significant restrictions following the 2007 Supreme 
Court rulings in Parents Involved in Community Schools v. Seattle School District. Race-neutral 
assignment plans, which consider factors other than race when attempting to achieve more socially 


!Race-conscious plans use race or ethnicity as a deciding factor in assigning students to schools. 
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balanced enrollments among schools, are more prevalent in today’s post desegregation world.” 
For example, the San Francisco Unified School District’s open enrollment program uses five race 
neutral factors—poverty, language, and achievement indicators among them—to make placement 
decisions for oversubscribed schools.* 

Choice programs that do not monitor the distribution of students are considered less regulated. 
Most open enrollment and charter school programs use unregulated arrangements (Holme & 
Wells, 2008). Some state statutes encourage charter schools to seek school compositions that are 
similar to those of the school district in which the charter school resides. In practice, however, 
these hortatory policies seldom lead to genuine attempts to achieve racial or social class balance 
(Holme & Wells, 2008; Wells, Holme, Lopez, & Cooper, 2000). The general intent of unregulated 
choice programs is to encourage market-based competition for students, thereby improving, it 
is assumed, school quality. Another aim is to provide alternatives for students attending low- 
performing schools. It is assumed that all families have the time, information, and resources to 
change schools if they wish. A primary objective of controlled choice plans, on the other hand, is 
to give priority to students from disadvantaged backgrounds. Controlled choice is also designed 
to prevent increased social stratification and to promote more integrated peer environments. 
School choice student assignment strategies, whether they are race-conscious, race-neutral, or 
unregulated, can produce very different patterns of student enrollments. 


PART |: SYNTHESIZING KEY EMPIRICAL FINDINGS 


Universal claims about the effects of school choice plans are difficult to defend, given that they 
operate in very different social, historical, legal, and political circumstances. Moreover, school 
choice schemes vary in their regulation of the student assignment process as well as in the 
provision of transportation. The studies discussed here, however, point to three empirical claims 
that receive support from a broad research base. 


Claim 1: Choice Assignment Programs That Ignore Students’ Race, Class, or 
Achievement Levels Run the Risk of Exacerbating Stratification 


Three studies in this issue provide evidence that directly support this first claim (Bifulco, Ladd, & 
Ross, 2009/this issue; Holme & Richards, 2009/this issue; Koedel, Betts, Rice, & Zau, 2009/this 
issue) and three other studies provide evidence for why and how it might be true (Bell, 2009/this 
issue; Saporito, 2009/this issue; Smrekar, 2009/this issue). 


2There is some evidence that race-neutral choice programs are successful at diversifying students socioeconomically 
(Mickelson, 2004); however, they fail to desegregate very well by race or ethnicity (Brief of the Council of the Great City 
Schools, 2006; Kahlenberg, 2001; U.S. Department of Education, 2003). 

3Conventional wisdom even among quite well-educated people holds that school desegregation as practiced in the 3 
decades following Brown v. Board of Education has failed—failed to make schools better, failed to close the achievement 
gap, failed to usher in an age of equality. These opinions are never grounded in solid evidence and may be more wishes 
than reality. It is the present stage in the struggle for racial equality in America that we label a “post-desegregation world.” 

4See http://portal.sfusd.edu/template/default.cfm?page=policy.placement.process. 
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Bifulco et al. (2009/this issue) take advantage of a unique dataset that allows the identification 
of the neighborhood schools to which choice students would have been assigned in the absence of 
choice. Forty percent of the elementary and middle school students in Durham, North Carolina, 
opted out of their assigned school and attend a magnet school, another zoned school, a year-round 
school, or a charter school. The greatest movement was observed among the more advantaged 
students who tended to “opt out of schools with concentrations of disadvantaged students to 
attend schools with high levels of achievement” (p. 147). Achievement-skimming most affected 
schools located in attendance zones with large (but not exclusive) concentrations of disadvantaged 
students. Distance was apparently a factor as “the closer these zones are to choice schools that are 
attractive to high achieving students, the larger the effects of choice on peer achievement levels” 
(p. 147). These findings—which imply that educationally advantaged students exercise choice 
more often, do so to leave disadvantaged settings, and opt for more advantaged schools—are 
consistent with investigations of choice programs in other cities such as Chicago (Cullen, Jacob, 
& Levitt, 2005); Charlotte, North Carolina (Hastings, Kane, & Staiger, 2005); and Cleveland 
(Campbell, West, & Peterson, 2005). 

Using student transfer data, Holme and Richards (2009/this issue) find that interdistrict choice 
separated students by race and socioeconomic class in metropolitan Denver. Students from higher 
income families were far more likely to participate in interdistrict choice, and overall, choice 
students were much more likely to transfer to wealthier school districts. White students were 
more likely to opt out of racially diverse districts and transfer to districts with greater percentages 
of White students. This last finding is consistent with evidence that Saporito (2009/this issue) 
reports here regarding increases in White enrollments in private schools in communities that 
became more racially diverse. 

In some cases, economically disadvantaged students of color in Denver used choice to transfer 
out of mostly White and wealthy districts to districts with greater percentages of children from 
their own backgrounds. This effect on school peer composition is consistent with what Bifulco 
et al. (2009/this issue) and others (Brunner, Imazeki, & Ross, 2006; Fairlie & Resch, 2002; 
Lankford & Wyckoff, 2006; Saporito, 2003) have reported, suggesting that some families of 
color want their children to attend schools with other students of color. The findings presented 
by Holme and Richards (2009/this issue) also resonate with parent choice studies reported here 
(Bell, 2009/this issue; Smrekar, 2009/this issue) and elsewhere (Clark, 1991, 1992; Gill, Timpane, 
Ross, & Brewer, 2001; Henig, 1990, 1996; Levin, 1999). These studies suggest that there exists 
a preference among parents for schools with peer compositions that look similar to their own 
child’s race and social status. 

Koedel et al. (2009/this issue) evaluate the integrative and segregative effects of three choice 
programs in the San Diego Unified School District, the nation’s eighth largest school system. The 
open enrollment program places no conditions (beyond space restrictions) on student transfers 
either within or outside of districts; does not provide transportation; and, among the district’s 
choice options, is the “closest to an unregulated market” (p. 118). Koedel et al. (2009/this issue) 
find that the open enrollment program exerted segregative pressures on the district in terms of 
race, socioeconomic status, and student achievement. By all available measures of socioeconomic 
status, disadvantaged students were underrepresented among those who changed schools. More 
advantaged students used choice to switch to schools of higher socioeconomic status, a finding 
consistent with the related literature (see, e.g., Armor & Peiser, 1998; Goldhaber, 1996). The 
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segregating effect on achievement was so significant that it overwhelmed the integrating effects 
of two other popular choice programs in the district. 

Common among the choice programs that increased stratification in Durham, Denver, and San 
Diego is the absence of policies based on race, class, achievement level, or language status that 
regulate student transfers. The programs represent laissez-faire, free market-oriented choice plans 
that provide limited or no funding for transportation. Some of Durham’s school choice programs 
do provide transportation, but since 1999, their assignment policies have been race blind and 
make no direct use of other individual student characteristics such as achievement or poverty 
(free-lunch status). 

The reasons unregulated choice plans demonstrate stratifying tendencies are explored here 
by Saporito (2009/this issue), Bell (2009/this issue), and Smrekar (2009/this issue). Saporito 
finds strong correlational evidence suggesting that parents—White parents, in particular—choose 
schools at least in part on the basis of race. This finding is consistent with research on other 
unregulated choice programs (see, e.g., Bifulco & Ladd, 2007; Cobb & Glass, 1999, 2001; 
Saporito, 2003). 

Bell’s (2009/this issue) study of a diverse group of 48 parents in a Midwestern city provides 
insight into why unregulated choice programs tend to stratify schools. Bell found that parents’ 
decision strategies appeared to be associated with social class status. Bell reports that, “middle- 
class parents’ social networks put them in contact with a higher proportion of non-failing [and] 
selective ... schools than did poor and working-class parents’ networks” (p. 202). 

Smrekar (2009/this issue) examined the reasons behind the shift in magnet enrollments from 
integrated to predominantly African American before and after the declaration of “unitary status” 
in Nashville, Tennessee. Her case study identified a complex array of influences on parents’ 
decisions about schools and emphasizes the importance of parent social networks. For White 
parents, information was primarily shared among parents of students attending a feeder preschool. 
For African American parents, information was predominantly passed through church and work- 
based social networks. Consistent with findings reported by Bell (2009/this issue), parents’ 
choices appear to be significantly shaped by social class networks and race. 

Choice policies that place the burden on families to exercise choice may run the risk of further 
socially stratifying schools. Studies in the United Kingdom have found that parent choice is 
systematically associated with social class differences (Ball, Bowe, & Gewirtz, 1996; Gewirtz, 
Ball, & Bowe, 1995; Reay & Ball, 1997). Here, Bell (2009/this issue) and Smrekar (2009/this 
issue) bring to the surface the implications of parent decisions that are associated with and greatly 
influenced by their own race and social class. Together these studies suggest that unregulated 
choice may contribute to class-based stratification of students as a result of the decision-making 
patterns of parents.° 

Two recent reviews of research corroborate the evidence presented here. The National 
Academy of Education (NAE)® concluded that choice plans that do not regulate student 


5To the degree that race and class intersect, parent choice-making patterns may also lead to greater racial stratification 
under unregulated schemes. Holme and Wells (2008) drew a connection between the lack of programmatic support 
provided to low-income families and their decisions. To them, “it comes as no surprise that virtually all evaluations of 
state interdistrict, open enrollment choice programs show that low-income students and students of color are the least 
likely to participate in these programs” (p. 177). 

©For this report the NAE convened the Committee on Social Science Research Evidence on Racial Diversity in Schools. 
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assignment in a race-conscious manner have the tendency to increase racial stratification (Linn & 
Welner, 2007). Similarly, Holme and Wells (2008) reported that, “laissez-faire open enrollment 
policies have led to greater racial and social stratification between districts” (p. 176). From the 
evidence presented here and elsewhere, the claim that unregulated school choice leads to greater 
stratification among students is made on substantial empirical footing. 


Claim 2: Controlled Choice Programs Have the Potential to Redress Minority and 
Economic Isolation 


Two of the programs examined by Koedel et al. (2009/this issue), the Voluntary Ethnic Enrollment 
Program (VEEP) and a magnet school program, have been in operation since the 1970s. The 
VEEP and magnet programs are designed to encourage students residing in White communities 
to attend schools in non-White communities, and vice versa. Both programs improved integration 
along indicators of race, achievement, and parent education level. In addition, VEEP integrated 
students by language status. Neither program is permitted to use race as an explicit criterion 
for admission, but VEEP is particularly race-conscious as it connects White and non-White 
neighborhoods with common busing routes. The magnet program uses socioeconomic clusters 
defined geographically, with preferences given to students living in clusters that look least like 
the magnet student composition. Unlike the open enrollment program in San Diego, VEEP and 
the magnet program provide transportation and regulate the assignment of students. 

In their comprehensive review of interdistrict school choice programs, Holme and Wells (2008) 
found that desegregation programs, in contrast to open enrollment plans, are the most effective 
at serving the least advantaged students. They reported that interdistrict school desegregation 
programs “led to greater diversity in the suburban districts and less racial isolation in urban 
schools” (p. 176). The NAE review of research on school choice and stratification, which was 
conducted just prior to the Seattle and Louisville cases heard by the Supreme Court in the summer 
of 2007, likewise concluded that controlled choice policies that oversee student assignment with 
race in mind have the best chance of reducing racial isolation (Linn & Welner, 2007). Political 
will, fiscal resources, and legal supports are required to support controlled choice programs that 
aim to serve the least advantaged students. 


Claim 3: Evidence Is Not Strong That Unregulated Choice Programs Consistently 
Provide Educational Benefits Either in Terms of Increased Student Achievement 
or Increased Organizational Innovation 


The theory of action underlying charter schools is that bureaucratic deregulation and competition 
will spur innovations in curriculum and instruction, which in turn will lead to higher student 
achievement. The articles by Huerta (2009/this issue) and Scott and Villavicencio (2009/this 
issue) in this issue examine the empirical support for this claim. 

Huerta (2009/this issue) followed the evolution of a grassroots charter school in California. He 
traces the challenges faced by a fully autonomous school and observed the “isomorphic pressures 
of the wider institutional environment” (p. 256) that ultimately undermined its independent status. 
The school, which started off as progressively innovative in a number of ways, began to adopt more 
traditional forms of schooling (some only symbolically) to gain external legitimacy. The evidence 
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presented by Huerta is consistent with the few studies that address the supposed innovative effects 
of charter schools (Henig & McDonald, 2002; Rofes & Stulberg, 2004; Schneider & Buckley, 
2003). His case study illustrates how innovations in school organization and instruction can occur 
as a result of reducing regulations, but that these changes are tempered by the institutionalized 
conditions that legitimize and preserve the status quo (see also Lubienski, 2003). Huerta points 
to the dearth of empirical support for the theory of action that deregulation leads to sustained 
organizational innovation. The point here is not that unregulated school choice programs, such as 
charter schools, fail to foster innovation but rather that there is not yet much evidence that they 
do spark innovation. 

In contrast, far more research on charter schools has focused on their effects on student 
achievement. An often cited America Federation of Teachers (AFT) report on charter school 
effectiveness, which prompted a heavy backlash from charter school advocates, concluded that 
charters’ effects on achievement are mixed at best. Other studies confirmed many of the AFT’s 
findings (Lubienski & Lubienski, 2006; U.S. Department of Education, 2004). In this issue, 
Scott and Villavicencio (2009/this issue) highlight the intense politicization of charter school 
effectiveness research and, after sifting through the evidence, conclude that the effects of charter 
schools on student achievement are indeed decidedly mixed (see Hill, 2005). More broadly, 
Lubienski, Weitzel, and Lubienski (2009) provided a searching analysis of how school choice 
advocates have attempted to portray empirical research on choice and student achievement as 
leading to a consensus: 


If there is a consensus on achievement in school choice plans, it is not the consensus that school 
choice advocates have attempted to fashion. In fact, the research on this issue suggests a much more 
complex picture, with mixed findings that generally indicate minimal advantages from choice in 
the best light. Yet patterns are inconsistent, with most groups typically seeing no gains, and even 
causation is unclear in the isolated instances where gains may appear. (p. 183) 


Scott and Villavicencio (2009/this issue) speculate that the racial and economic composition of 
student populations in certain charter schools may intersect with the academic outcomes they pro- 
duce. Not unlike other schools, charter schools that serve poor students of color in isolation, for the 
most part, struggle to produce good academic outcomes. There are notable exceptions, of course. 
However, strong performance in exceptional cases may be because of selective admissions criteria, 
parent contracts, and other criteria that may serve to exclude the more difficult to educate students. 

The educational implications for students in charter schools that promote integrative or seg- 
regative environments are potentially significant. Unregulated choice programs, such as charter 
schools, appear to lead to more race and class stratification (Claim 1). Unregulated programs 
that increase segregation among students, particularly low-income students of color, potentially 
foster all the challenges associated with those isolated environments (lower teacher quality, fewer 
resources, and the like; see, e.g., Brief of 553 Social Scientists, 2006; Mickelson, 2004). 


PART II: INFORMING POLICIES ON SCHOOL CHOICE 


Judgments about the importance of reducing segregation and stratification depend both on em- 
pirical evidence about the effects of segregated schooling on individual’s lives and on normative 


SCHOOL CHOICE IN A POST-DESEGREGATION WORLD 269 


arguments about what values ought to guide government actions. The articles here do not directly 
address either of these issues. Here we offer some reasons for believing that the amount of segre- 
gation and stratification does have important consequences for the lives of students, particularly 
underprivileged students. We also offer a view that the potential benefits for economically ad- 
vantaged White students and poor students of color from reducing segregation and stratification 
is sufficient to warrant continuing efforts to promote integration. 


Integration Policy and Human Capital 


Integration policies are intended to bring together people of different backgrounds to learn from 
each other and benefit from diverse cultural perspectives. Integrated settings provide otherwise 
economically and racially isolated students access to diverse knowledge and information net- 
works. These networks can lead to opportunities for success in postsecondary attainment and in 
the labor market. 

The peer environments of schools contribute to the development of students’ education 
capital—a form of human capital (Becker, 1964/1993). Human capital is a tacit resource of 
unique knowledge and skills that can be exploited to produce an outcome or personal gain. 
Education capital is a resource that, when expended, leverages learning and adaptability to new 
contexts. Children and families possess their own distinctive forms of cultural knowledge, and 
they live and communicate within particular social networks. Through the transmission of cultural 
and social capital, peers influence one another’s academic attitudes, achievement, knowledge of 
career paths, cultural competence, and lifetime trajectories. 


Cultural and Social Capital. Cultural capital represents the “forms of knowledge, skills, 
education, and advantages that a person has, which give them a higher status in society” (“Cultural 
Capital,’ 2008, para. 2; see also Bourdieu, 1986). Bourdieu noted that parents provide their 
children with cultural capital “by transmitting the attitudes and knowledge needed to succeed in 
the current educational system” (“Cultural Capital,’ 2008, para. 2). Such knowledge transmission 
occurs through other sources as well, including interactions with peers, adults, and the popular 
media. Cultural capital may be one of the key avenues through which race and class-based 
structural inequities are reproduced by schools. Cultural capital is bequeathed, or passed on, 
within intact social groups. It acts as “a social relation within a system of exchange that includes 
the accumulated cultural knowledge that confers power and status” (Barker, 2004, p. 37). Cultural 
capital is an important resource that schools can develop within their students. Students possessing 
the cultural capital necessary to succeed in schools know which education levers to pull and which 
doorways lead to opportunities. Schools with high levels of this type of cultural capital are well 
positioned to provide instrumental and intrinsic benefits on children.’ 


7Harry Brighouse’s (2000) notion of education equality emphasizes “the benefits that education provides opportunities 
for,’ which he characterizes as either instrumental or intrinsic (p. 117). Instrumental benefits are those that contribute to 
success in the labor market while intrinsic benefits are those that contribute to a more fulfilling life beyond work (see 
Brighouse, 2000, p. 121). 
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Harvard political scientist Robert Putnam (2000) noted that “child development is powerfully 
shaped by social capital” (p. 296). He differentiated bonding social capital from bridging social 
capital. Bonding represents the strength of social networks within homogeneous groups and 
bridging refers to the value of social networks between homogeneous groups. Bridging forms of 
social capital are thought to bring about benefits to the groups involved and to society at large. 
Social networks represent a form of social capital, and are one of the more valuable resources one 
encounters in schools. Through peer networks, students acquire academic norms, the ability to 
cooperate and compete, and they learn from each other. The bridging of social networks among 
students from diverse backgrounds can most easily occur in integrated settings. Children from 
disadvantaged backgrounds can readily benefit from bridging to new social networks that have 
the potential to improve their chances of succeeding in life beyond school.’ Likewise, White 
students from more privileged backgrounds can benefit from the sociocultural awareness and 
sensibilities of urban youth of color. 

The value of an individual’s cultural or social capital depends on the context within which it 
is used. Consider formal schooling, a context legitimized by dominant social groups. Many inner 
city children from uninformed, socio-economically disadvantaged families do not have access to 
the information and knowledge networks that would lead them to tertiary education. Compare 
this to a suburban child from a wealthy, college educated family; for this child the pathway to 
college has likely been there from the start and is only further reinforced by a school serving 
children with this same advantage.” 

School choice plans redistribute students possessing varying degrees, and different types, of 
cultural and social capital. These forms of capital represent important resources for schools and 
students. Integrated schools offer children more diverse sets of cultural and social capital than do 
segregated schools. Schools isolated by minority race or poverty are more likely to possess forms 
of capital that have limited utility in the spaces legitimized and protected by the dominant social 
class.!° 


Other Forms of Integration Promoting Cultural and Social Capital. One might specu- 
late that the transmission of cultural and social capital to the least privileged children stands a 
better chance of taking place in integrated school settings than in segregated ones.'! But stepping 
back from questions of education policy helps to provide a broader view that is more complex 
and, perhaps, more promising. Schools represent one venue for the exchange of cultural and 
social capital, but so too do the communities in which children, families, and adults live. Con- 
centrated poverty in schools is a reflection of concentrated poverty in neighborhoods. Studies 


8There are, of course, potential costs associated with integrating students of color into more “White” environments, 
namely, as some would argue, they are being asked to assimilate to dominant cultural norms at the expense of their own 
identities. As researchers, we are still grappling with these complexities. 

Certainly knowledge of tertiary institutions, or even information on how to access them, cannot equal the leverage 
associated with legacy and wealth; racism still pervades these spaces. 

10-Yosso (2005) challenged the notion that “People of Color ‘lack’ the social and cultural capital required for social 
mobility” (p. 70). We want to stress that all schools and students carry high levels of cultural capital; however, some 
forms are more valued than others depending on context and who makes the determination. 


11 For a brief review of research on network theory, or how social capital translates into outcomes, see Linn and Welner 
(2007, pp. 29-32). 
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of two important social experiments—the Gautreaux Assisted Housing Program and Moving 
To Opportunity (MTO)—point to housing integration as a potential avenue for expanding the 
cultural and social capital of disadvantaged groups. 

As a result of a lawsuit filed in 1966 by Dorothy Gautreaux alleging discrimination in public 
housing, the Chicago-based Leadership Council for Metropolitan Open Communities, a private 
nonprofit fair housing organization, contracted in 1976 to run a program of relocating poor African 
American families (many of them single parent) to middle-class and more affluent White suburbs. 
The program was closed in 1998 after meeting a goal of relocating more than 7,000 families. 
The fate of these relocated families (suburb movers) became the subject of study by several 
social scientists, among them Kaufman and Rosenbaum (1992) and Rubinowitz (1992). Families 
relocated to more affluent White suburbs were generally satisfied with the program, praising the 
safety and greater quality of their children’s schools. When suburb movers were compared with 
similar Gautreaux children who were relocated to predominantly African American urban areas 
(city movers), their superior performance in school and afterward was apparent. Suburb movers 
were more likely to be enrolled in high school, to be in a college prep track in school, to be 
enrolled in a 4-year college, or employed some 8 to 13 years after being relocated. Significantly, 
these benefits arose merely from relocation and not from the provision of additional services or 
assistance from the program. 

In 1994, the Department of Housing and Urban Development launched a randomized experi- 
ment (MTO) on the effects of relocating poor, primarily minority families from highly distressed, 
urban neighborhoods to low poverty neighborhoods (Ludwig, Duncan, & Hirschfield, 2001). 
“Experimental” group families received housing subsidies and search assistance to move to low- 
poverty census tracks. A second treatment group received subsidies to relocate but no assistance 
in locating desirable neighborhoods; a third group, Control, received no attention from the pro- 
gram. Estimated impacts on young people in the MTO experimental group were much more 
mixed than in the Gautreaux studies. Early investigations indicated that MTO reduced antisocial 
behavior by youth in Baltimore and Boston, but not in New York (Katz, Kling, & Liebman, 1999; 
Kling, Ludwig, & Katz, 2005), and later follow-ups found fewer violent crime arrests among the 
experimental group. There were also strong, positive impacts on the physical and mental health 
of adult and adolescent females (Ludwig et al., 2008). Two or 3 years into the program, however, 
property crimes among MTO males went up relative to controls, and other behavioral effects on 
adolescent boys were not so positive. Perhaps most relevant here, a 5-year follow up study of 
all five MTO sites reported minimal overall impact on participants’ employment and education 
outcomes (Orr et al., 2003); however, the lack of effects on education outcomes may be because 
most MTO children remained in their original (isolated) urban schools. 

The Gautreaux study and the MTO experiment shed light on the potential effects of integrated 
environments on underprivileged groups. Further, they remind educationists to explore other ways 
to bring about increased integration and to enhance the cultural and social capital of children and 
their families. 


Normative Arguments Favoring Integration and Their Implications for Policy 


Empirical evidence and normative argument are two foundations on which judgments about 
education policies can be made. Both sources inform education policy. Normative reasoning brings 
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meaning to empirical claims by revealing which values are worth pursuing through government 
policy. The justice of an education policy is arguably the most important criterion for policymakers 
to consider when choosing among alternatives (Rallis, Rossman, Cobb, Reagan, & Kuntz, 2007). 
Next we draw on the philosophical ideas of John Rawls. 

In A Theory of Justice, Rawls (1971) published his attempts to reconcile the inherent tension 
between liberty and equality—two competing values at the heart of the school choice debate. 
Individual freedom of choice can be at odds with equality of education opportunity, and vice 
versa. Central to Rawls’s justice-as-fairness doctrine is the maximin principle. That is, to address 
social and economic inequalities, government actions must confer the greatest benefit on the least 
advantaged members of society.!? For Rawls, social justice is about enhancing the life chances 
of children who start in disadvantaged positions. 

Education policy should be concerned first and foremost with the effects on those worse off. 
Poor students of color, particularly those in isolated environments, tend to be among the least well 
off in the U.S. education system.!? In the United States, education quality is highly correlated with 
community income. Higher wealth communities tend to have more resources, better facilities, and 
better schools. In contrast, many of our urban centers suffer from inadequate education resources 
and facilities and overall lower quality schools. Residential and school segregation by class and 
race is widespread in many of these urban settings. Significant numbers of poor people of color 
live and go to schools in isolation from wealthier citizens on the other side of town. There are, of 
course, pockets of success, but generally speaking, poor students of color who attend schools in 
isolation receive relatively inferior educations (Brief of 553 Social Scientists, 2006; Mickelson, 
2004). 

Education policies that exacerbate the isolation of poor students of color are likely to make 
these populations of students worse off. Unregulated choice programs make it more difficult for 
students of color to attend schools with more diverse peer environments. For students attending 
economically and minority isolated schools, access to new forms of human capital is significantly 
restricted. Policies that put these students at such a disadvantage are socially unjust according to 
Rawls’s principle of fairness. 

In comparison, regulated choice programs have the potential to reduce minority and economic 
isolation and to promote diverse school environments. Diverse peer environments expose students 
to new social networks and forms of cultural capital that can expand opportunities for success in 
school, college, and the labor market. Students from isolated settings may particularly benefit from 
acquiring access to what amount to hegemonic forms of knowledge and information—defined 


!2Brighouse (2000) elaborated Rawls’s theory of social justice for school choice. Brighouse claimed that three condi- 
tions must be met to achieve full equality of education opportunity. First, the quality of a child’s education should not 
reflect parents’ wealth. Second, the quality of a child’s education should not reflect parents’ ability to choose schools 
for their child. Third, students with educational disabilities should receive more resources than nondisabled students. 
Brighouse’s last principle is conceptually similar to Rawls’s maximin principle. Providing relatively more resources for 
students with disabilities is in the spirit of achieving greater equity. Recasting the principle as providing relatively more 
resources to students with higher needs (more generally) implicates students from economically and socially disadvan- 
taged backgrounds, who tend to have higher needs than more socioeconomically advantaged students. It is not clear, 
for example, how or why one would distinguish a physical handicap from an economic handicap in Brighouse’s line of 
argument. 

!3 Seventy-five percent of people living in high poverty communities are people of color (Powell, 2001). Brooks-Gunn, 
Duncan, and Mariatato (1996) reported a significant proportion of people of color tend to grow up and continue to live in 
poor conditions. 
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by and reserved for dominant societal groups. These forms of capital represent an offshoot of 
the “hidden curriculum” that exists in schools with more privileged status.'* In a similar vein, 
housing integration policy has the potential to disrupt the cycle of poverty among those living in 
economically distressed communities. 

There is little evidence that the effects of increased isolation of poor students of color (brought 
on by unregulated choice plans) are likely to be offset by other benefits of school choice. 
Unregulated choice programs have not been proven to effectively improve academic achievement 
or result in the type of innovations that would lead to improved achievement. Therefore, in 
designing ways to expand parental choice of schools, policymakers should find ways to minimize 
any segregating effects choice might otherwise produce, and where possible try to find ways to 
use choice to promote integration. 


RECOMMENDATIONS 


The effect of policies on the least well off children is of paramount importance. School choice 
policies, in particular, should be evaluated against the evidence of their impact on those students 
who are disenfranchised.!° Acting on these principles, policymakers should resist adopting fiscally 
attractive, unregulated choice programs that further the segregation of students and that lack 
evidence of academic success with poor students of color. It also follows that policymakers should 
pursue choice policies that diversify peer environments to improve the education opportunities 
for disadvantaged students, in particular. 

School choice policies that regulate student assignment offer the best chance to integrate 
schools. Programs that oversee student assignment with race in mind are the most effective at 
reducing racial isolation (Linn & Welner, 2007). The NAE concludes that 


school choice generally, and magnet schools in particular, do have some potential to reduce racial 
isolation. However, that potential is only likely to be realized to any significant degree if enrollment 
constraints ...are part of the school choice policy. Without such constraints, school choice plans 
appear to contribute to segregation more often than they ameliorate segregation. (Linn & Welner, 
2007, p. 40) 


The most equity-minded interdistrict plans are controlled choice plans (see Holme & Wells, 
2008). Voluntary interdistrict desegregation programs “ensure that students are provided infor- 
mation, transportation, and support in their chosen school” (Holme & Richards, 2009/this issue, 
p. 170). 

Although the Parents Involved y. Seattle decision places limits on race-conscious assign- 
ment policies, Justice Kennedy’s concurring opinion offered justification for controlled choice 
programs that treat race as a priority: 


If school authorities are concerned that the student-body compositions of certain schools interfere 
with the objective of offering an equal educational opportunity to all of their students, they are free 


14These hegemonic forms of capital (what we have referred to as a hidden curriculum) are also exploited by privileged 
groups within certain schools. 
15We stress that any impact should be judged on not only choice participants but nonparticipants as well. 
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to devise race-conscious measures to address the problem in a general way and without treating each 
student in a different fashion solely on the basis of a systematic, individual typing by race. 


Justice Kennedy also offered several specific ideas for constructing race-conscious assignment 
schemes (see concurring opinion in Parents Involved v. Seattle). 

Not all choice programs that are designed to desegregate consistently produce the intended 
outcomes. For instance the most recent evaluation of the federal Magnet Schools Assistance 
Program found that magnet programs achieved variable success in terms of integration (U.S. 
Department of Education, 2003). The effects of voluntary desegregation programs, such as 
magnets, on school racial and socioeconomic composition appear to depend on specific contextual 
conditions (demographic, legal, historical). Which conditions lend themselves to more or less 
racial integration continues to be the subject of research.!© Many of the studies in this special issue, 
in fact, examine the relationship between system conditions and changes in school compositions. 

There is only so much that regulatory education policy can accomplish. The regulation of 
student assignment within choice schemes does not ensure that the system will achieve racial 
balance, or prevent others in the system from seeking school placements that run counter to 
the intentions behind the regulation. Regulation cannot provide more seats for oversubscribed 
schools or create more demand for undersubscribed schools. It cannot shorten bus rides for 
students seeking choice options several miles or even towns away. Over the past few decades 
education policies designed to decouple residential and school segregation have achieved modest 
success; however, such efforts have been curtailed by a recent reversal of desegregation orders 
across the nation (Orfield & Lee, 2007). And, as noted earlier, voluntary school desegregation 
efforts are under even more scrutiny following the Supreme Court’s Parents Involved vy. Seattle 
decision. 

Perhaps most important, housing policy should be considered alongside school desegregation 
to ameliorate the inequitable conditions brought on by concentrated poverty that exists both 
within schools and the neighborhoods they serve. In his testimony to the National Commission 
on Fair Housing and Equal Opportunity, Gary Orfield makes the appeal. 


If we are not going to do anything serious about integrating schools, which is what the Supreme 
Court has told us in the Louisville and Seattle decisions last year, we absolutely have to get serious 
about housing integration, and we have to think very seriously about how to help the interracial 
communities in the United States and their schools deal with the interracial conditions successfully 
and remain integrated. (Testimony of Gary Orfield, September 8, 2008) 


Researchers of school choice can also be guided by Rawls’s theory of justice. They are urged 
to pay particular attention to a choice program’s effect on the least advantaged students in the 
system. As Brighouse (2000) argued, it is not only the consequences to the disadvantaged but their 
relative position in the system that matters. Studying the impact of a choice program on academic 
achievement would mean not simply looking at the average achievement levels among students in 


16Of interest, the NAE report draws a distinction between “racial diversity” and “racial integration.” Racial diversity 
refers to the racial composition of the entire school prima facie. In contrast racial integration represents positive intergroup 
contact and relations within those settings. We have learned that just because a school is diverse on paper does not mean 
there is not within-school stratification through a tracked curriculum or ability grouping that happens to segregate students 
by race and class. 
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a choice school or a choice program, but rather examining academic performance among students 
from all social classes or from all racial groups in the greater system. Choice affects not just those 
who participate but nonchoosers as well.'’ Researchers should provide policymakers with big 
picture accounts of how choice programs impact education systems. Researchers of school choice 
are encouraged to continue to assess the effects of choice on peer environments and the effects 
of these peer environments on students. 


CONCLUSION 


The articles in this special issue cover a range of topics, including the effects of school choice 
programs on organizational innovation and student achievement. Many of the authors examine 
the consequences of particular choice schemes on the social composition of schools, or the larger 
systems in which they are situated. The introduction of choice plans in any locality alters the 
racial and socioeconomic distribution of students across schools. Unregulated choice schemes 
tend to exacerbate stratification, and some regulated choice plans have the potential to diversify 
schools, or at least mitigate the effects of segregation. 

We have argued that education policy should be judged by the degree to which it benefits 
or does no more harm to the least advantaged students. Choice policies that do not prevent 
stratifying effects are socially unjust because disadvantaged students are placed in even more 
disadvantaged positions.'® Their education opportunities are set further behind those of their 
more advantaged peers who are clustered in more privileged settings. In contrast, choice policies 
that pursue integration have the potential to improve the lives of underprivileged students. 
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Introduction to the Special Issue on the New Politics 
of Educational Leadership 


Dorothy Shipps and Judith Kafka 
Baruch College, CUNY 


Two decades ago the Politics of Education Association published a yearbook describing 
changes in school administration. It predicted a new era of top-down educational policy making, 
the separation of educational leadership from the role of the principal, and changes in collec- 
tive bargaining to reflect demands for professional participation and discredited managerialism. 
Written in the midst of a period of highly touted school reform, these predictions were bold and 
prescient, but inevitably missed a great deal. 

At the close of the 21st-century’s first decade, the authors assembled here not only agreed 
to tackle these issues with the benefit of hindsight but also risked some predictions of their 
own. Collectively, they present evidence that educational leadership—a term that encompasses 
political expectations as well as the managerial and instructional responsibilities long associated 
with administration—has been extended beyond formal roles like the principalship, although 
principals remain firmly at the center. Educational leadership has expanded to encompass top- 
down decision making by a range of executive politicians and new-style district leaders who 
eschew educational credentials. At the same time, changes in the institutional environment of 
schools have required principals to pay greater attention to external constituents and market forces 
even as micropolitical studies of school leadership identify many variations. The five articles in 
the first half of this issue examine power, authority, and decision making in these political and 
administrative roles. 

The second half of the issue examines some of the institutional actors that are influencing 
leaders. Each of these five articles addresses the privatization of political influences in education 
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and asks us to reexamine our assumptions about how the politics of education should be studied. 
Some forms of privatization have spurred external actors to take on internal governance and 
leadership functions without the public accountability associated with traditional roles. Others 
function more like powerful interest groups that complicate, but do not fundamentally alter, lead- 
ership. Many have found ways to affect policy-making processes, as do some union presidents, 
district-level partnerships, educational management organizations, charter schools, and nontradi- 
tional leadership preparation programs. Collectively, such variations on the theme of privatization 
complicate our notions about what constitutes political action and political leadership in schools 
and districts. 

Jeff Henig opens the first section by asking the question, Have a new breed of education 
executives—presidents, governors, and mayors—usurped the legislative policy-making processes 
once perceived to be dominated by elected school boards, legislatures, and congressional commit- 
tees? By presenting multiple explanations of executive officials’ contemporary prominence, he 
suggests that interpretations relying on local history, visionary leaders, and transitory events are 
incomplete. Structural and institutional changes, he concludes, are driving a once-in-a-century 
reintegration of educational politics back into the arena of general-purpose politics. 

Elizabeth Useem tackles recent changes in the urban superintendency. Her archetype is Paul 
Vallas, currently the turn-around CEO of the New Orleans Recovery District, who served in the 
same position in Philadelphia between 2001 and 2007 and pioneered the role of noneducator 
turned top district leader in Chicago in 1995.' Focusing on his middle years in Philadelphia, 
she argues that Vallas pragmatically coopted the complex politics of market-based reforms while 
abetting the centralization of government authority. More positive than critical, she nevertheless 
points out that even an unusually talented, politically savvy individual who takes full advantage 
of policy loopholes cannot surmount the very real resource limits that urban districts have faced 
for decades. 

Judith Kafka fills a gap in our understanding of the principalship by summarizing its histori- 
ography. Notwithstanding the relatively skimpy historical literature, she debunks any lingering 
myths about the principalship having ever been an uncomplicated senior management or instruc- 
tional position. Instead, she notes that the principalship has always been multifaceted and deeply 
political: Among other things, the principal is accountable to a community constituency that has 
not always been even-handed in its assessment of the principal. She speculates that if anything is 
changing, it is the political context in which principals now work, as it may include an increase 
in market-based actors. 

Joseph Flessa takes up the contested terrain of school micropolitics by asking the question, 
Why has the research literature on distributed leadership in schools failed to address the literature 
on the micropolitics of the principalship, and vice versa? Not only are academic incentives for 
subfield dialog largely absent in education, he concludes, but also the normative and prescriptive 
nature of the writing on distributed leadership conceptualizes conflict as a pathology to be 
avoided. Conceptual boundaries are also constructed as problems to be ignored. The alternative, 
he suggests, is to reframe internal school conflict as one indication of what educators and other 
stakeholders value. 


'TIn confirmation of Henig’s argument, Vallas recently mused that he might leave New Orleans to run as a Republican 
for Cook County Administrator in Ilinois at the end of the 2008-09 school year. 
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Dorothy Shipps and Monica White argue that institutional changes intended to alter behavioral 
incentives in big city school districts like New York’s are already changing the politics of 
the principalship. Prior to a system of high-stakes accountability policies, which resulted in 
significant consequences for schools and their leaders, their research found principals enacting 
accountability in micropolitical terms: crafting an internal vision based on a professional and 
moral consensus and buffering it from outside interference. After high-stakes policies were 
implemented, they found principals had more or less realigned their professionalism with external 
influences: the district’s bureaucratic requirements, parents’ and politicians’ expectations, and the 
market imperatives of competition. 

Part 2 of this issue begins with Susan Moore Johnson and colleagues’ unique study of variation 
in the educational leadership displayed by local union presidents. Notwithstanding predictions 
of centralization within the two national teaching unions and across contracts, Johnson and 
colleagues found that local union presidents use the resources of their national affiliates but do not 
feel constrained by them. Local leaders adopt a range of policy tools beyond ritualized collective 
bargaining, and some outpace national affiliates in their embrace of reformist agendas like merit 
pay and peer review. Local variation creates opportunities for innovation and experimentation and 
is also personal: It is constrained by the relationship between the superintendent and the union 
president. 

Meredith Honig’s research on the political strategies used by external organizations to influence 
district policy in Oakland (California) and Chicago offers distinctions between those partnering 
organizations that adopt outsider and insider approaches. She concludes that external actors who 
stick to traditional influence strategies—leveraging foundation dollars, setting high standards for 
grant giving, and monitoring performance through progress reports—may be unable to sustain 
momentum or change educator’s perspectives. But external organizations whose leaders are 
willing and able to take responsibility for tasks that would otherwise be performed inside the 
district may have more influence and be better able to sustain the efforts they catalyze. 

Luis Huerta and Andy Zuckerman are also interested in the ability of organizations—in their 
case, boundary-blurring charter schools—to sustain innovation. They ask whether the instruc- 
tional and organizational changes initiated by charter schools can persist in the face of the 
taken-for-granted institutional scripts of schooling that confer legitimacy. If so, they suggest, it 
will be because their technical changes to instruction and organization survive long enough to 
alter the institutions that support and oversee them. 

Janelle Scott and Christine DiMartino argue that our understanding of the politics of privati- 
zation lags behind our knowledge about its individual and societal outcomes. They propose that 
privatization be reconceptualized as a series of internal and external roles that individuals and 
organizations adopt to refocus schooling away from traditional bureaucratic means of performing 
~ routine functions. Some of these roles, like profit seeking in public education, have a long history, 
but others, like ideological gate keeping, are relatively new. Nevertheless, as their application of 
the typology to New York City demonstrates, no abstract simplification reduces the complexity 
of the changes urban school systems are undergoing in the name of privatization. 

James Cibulka wraps up this section with an article on the new politics of accredita- 
tion. He compares the traditionally weak influence of accrediting organizations like NCAATE 
on college-based leadership preparation programs to its current influence given that nontra- 
ditional leadership preparation alternatives abound. In the past, college-based programs had 
little incentive to focus on outcomes-based standards. But he asserts that the new pluralist 
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political environment is having its desired effect: Colleges now experience a greater urgency to 
reform. 

Taken together, these 10 articles encourage speculation about privatization and political cen- 
tralization. If this assemblage of suggestive research and conceptual reframing is confirmed, 
long-standing border wars between centralization and decentralization in education governance 
may be overtaken by clashes between political executives and the forces of privatization. The 
boundaries between districts and their external partners in the 21st century will defy traditional 
definitions. New quasi-public governing roles will become institutionalized. The multiple re- 
sponsibilities of principals and superintendents will increasingly demand political skills and 
resources. 
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Various accounts have noted a growing visibility and influence of presidents, governors, and mayors 
in the shaping of educational policy. Typically, explanations for this are discrete and situational: 
specific to particular levels of government, particular jurisdictions, particular visionary leaders, and 
transitory external events. This article reviews the historical manifestations of growing executive 
involvement in education throughout the federal system and introduces four rivals to the situational 
perspective. The emergence of the so-called new educational executives, it concludes, grows out of 
structural shifts in the institutional landscape and may be best understood as the more visible front 
edge of a broader reabsorption of education politics into the arena of general-purpose governance. 


First came the education governor. In the early 1980s attention was paid to a handful of 
governors who were staking political capital on efforts to upgrade public education systems as 
a necessary component of economic development. On the heels of the 1983 A Nation at Risk 
report, James Hunt, who had made education his signature issue during his tenure as governor of 
North Carolina, issued a strong call to action to his colleagues gathered at the National Governors 
Association (NGA) annual meeting, in Portland, Maine. Speaking about education he said, 


We urge you to make this your top priority. To get your people interested. To get them educated in the 
matter. To develop a long-range plan for your state, but also your short-range legislative programs. 
So that is the first recommendation. A plan that you lead the development of education in your state. 
(Hunt, 1983, p. 7) 


His call apparently did not go unheeded. Two years later, an article in Education Week observed, 
it was “the rare governor who has not, in the past two years, devoted a large portion of his or her 
time, energy, and political capital . . . to nuts and bolts questions of education policy” (Caldwell, 
1985). 

Not long afterward, the term “the education president” came into vogue. Ronald Reagan had 
come into office calling for eliminating the U.S. Department of Education. Looking to succeed 
him, however, his vice president, George H. W. Bush, began his standard stump speech in the 1988 
campaign by declaring “I want to be the education president” (Hoffman, 1988). His successor, 
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Bill Clinton, rode into office on the backs of different issues (“It’s the economy, stupid’), but 
as a former “education governor” he came to the White House with strong views on education 
and steered policy in support of the growing standards movement and attempted to articulate a 
nonmarket rationale for charter schools. George W. Bush did not work hard to borrow his father’s 
“education president” language, but he, like Clinton, was able to claim expertise based on his 
prior experience as governor. His pro-education stance in the 2000 election dramatically reduced 
the traditionally huge Democratic advantage in the public’s view of which party would do a better 
job of ensuring educational quality (Hess & McGuinn, 2002).! “Most observers—whether they 
like it or not—regard it as the Bush Administrations’ signature domestic achievement, aside from 
tax cuts” (Hess & Finn, 2007, p. 2). 

In the meantime, elected executives were coming more into focus at the local level as well. 
During the 1990s, four major cities opted for mayoral control, instead of separately elected school 
boards: Boston (1992), Chicago (1995), Cleveland (1998), and Detroit (1999). Momentum built 
slowly but accelerated after the turn of the century, with Harrisburg and Philadelphia, Pennsyl- 
vania; Oakland, California; Providence, Rhode Island; Washington, DC; and, most prominently, 
New York City among those that have recently moved to mayoral control to at least some degree 
(Henig & Rich, 2004; Wong, Shen, Anagnostopolous, & Rutledge, 2007). 

Journalistic accounts often exhibit a “great man” perspective that heavily emphasizes the 
political vision and skills of the particular elected executives who spotted or created opportunities 
overlooked by their predecessors. This perspective understates the importance of the broader 
political institutional landscape, and the ways in which shifts in the views and power of other 
powerful interests helped to determine whether and when executive muscle could be effectively 
exercised. Even scholars that have given due weight to institutional and political contexts within 
which leaders operate have tended to treat the three movements as distinct, each accounted for 
by elements specific to the level of government, the political venues in which they are located, 
and the nature of the pressures and opportunities presented by conditions at the time. In this 
article I consider the possibility that the new role in education policy of presidents, governors, 
and mayors should be conceptualized instead as manifestations of more general phenomena. 
After elaborating on the historical manifestations of growing executive involvement in education 
throughout the federal system, I contrast the implicit theory that these are discrete and situational 
(opportunistic), with four specific rival explanations, each with differing implications about how 
widespread and how permanent this movement toward educational executives is likely to be. 

The emergence of so-called new educational executives, I conclude, has less to do with 
the characteristics of the leaders themselves, than with shifts in the political and institutional 
landscape. In place of a “great man” perspective, understanding the emergence of elected 
executives in educational governance requires consideration of shifts in the relative power and 
involvement of a network of actors who support or oppose such a change. Less obvious, but 
ultimately more important, these shifts in the institutional landscape have less to do with the 
power of the executive office, relative to the other branches of government, than with erosion 
of boundaries that have separated education politics from general purpose politics. Education 
has long been regarded as—and to a considerable degree functioned as—a decision-making 


'The Democrats had a poll advantage averaging more than 22 points on this question from 1979 to 2000. In 2000, 
this dropped to 3 points (Hess & McGuinn, 2002, Table 2). 

Indeed, in some instances the shifts in visibility and power are within the executive branch, from local and state 
superintendents to mayors and governors for example. 
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FIGURE 1. Trends in attention to elected chief executives and education.? 


arena buffered from the pull and tug of normal electoral and interest group politics (Wirt & 
Kirst, 2001). The ascendance of presidents, governors, and mayors, I conclude, likely constitutes 
the more visible front edge of a broader reabsorption of education politics into the arena of 
general-purpose governance, with implications that go beyond the executive office 


MAPPING THE TRENDS 


Figure | shows the trend in attention to the growing role of executives in education by mapping 
the uses of three terms in academic and general interest articles from 1980 through 2007. The 
search terms used were “education president,” “education governor,” and “mayoral control” plus 
“schools.’4 Prior to 1987, these terms rarely appeared in article titles or abstracts. 


3The search terms used were “education president” “education governor” and “mayoral control” plus “schools.” 

4Nothing I say here should be construed to indicate that governors, presidents, and mayors paid no attention to 
education prior to this. In calling for Congress to rename the U.S. Department of Education building after Lyndon Johnson, 
Democratic Congressman Gene Green, in March 2007, said, “Lyndon Johnson’s first priority in life was education, and 
he was the first education president” (Mittelstadt, 2007). Prior to the Progressive Reform Era, mayors played a formal role 
in running school systems in many urban areas, in some places that formal role never was superceded, and even when it 
was—as discussed later in this article, some mayors episodically used their informal powers to be involved with schools 
to a significant degree. In highlighting these recent movements I am drawing attention to the fact that these represented 
self-conscious and broadly manifested efforts to nonincrementally alter formal and traditional governance regimes. 

>The terms appear much more frequently (approximately 4 to 5 times as many articles) when one includes document 
text, but the timing of their appearance is the same. Even the handful of mentions highlights the extent to which executive 
involvement was something of a nonissue. For example, the 1982 reference to mayoral control was a New York Times 
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Governors 


Although it was not until 1987 that the term “the education governor” began to appear prominently 
in some articles, a marked change in the prominence of education as a focus for governors was 
evident well before that. Krotseng (1987) suggested that the concept of “education governors” 
should be traced to several early 20th-century Southern governors, such as Charles Brantley Ay- 
cock, who in assuming office vowed to “devote all four years of my official time to upbuilding the 
public schools of North Carolina” (p. 5). Maris Vinovskis mentions Andrew Jackson Montague, 
Virginia governor from 1902 to 1906, and Indiana’s Paul McNutt, who worked to increase state 
funding to fiscally challenged local districts during the Great Depression (Vinovskis, 2008). But 
it was in the early 1980s that the governors began to move on this front in a way that suggests 
that something systemic was going on, and not just the idiosyncratic initiatives of a few outlier 
pioneers. 

That Southern governors like James Hunt, Dale Bumpers, and Bill Clinton were prominent 
figures on this issue seems to reflect a realization, on their part, in the 1970s and 1980s that 
economic development could not succeed solely on the base of low taxes, low regulation, and anti- 
union laws. These sufficed initially to draw some industrial jobs from the north. In an increasingly 
global economy, however, even previously booming parts of the Sunbelt were beginning to 
experience corporate flight. “By 1980-81, unemployment rates in many of the more industrialized 
southern areas stood at above 10 percent” (Bluestone & Harrison, 1982, p. 39). It became apparent 
to at least some of the more foresighted southern leaders that long-term growth depended on 
investment in both infrastructure and schools. Not all governors or gubernatorial candidates 
who championed education investment were successful. The education governors had to wrestle 
with voters and legislatures that were often deeply resistant to the requisite tax increases and 
the infringement on local prerogatives. As the national economy seemed to falter, with serious 
recessions in both the early and late 1980s, the view of education as a weapon against global 
competition found a wider audience. 

Casual histories tend to allocate almost exclusive credit to A Nation at Risk for sparking the 
wave of national concern about an education “crisis” and the school reform and accountability 
initiatives that followed. This top-down perspective puts the White House in the foreground, but it 
is inaccurate to imagine that governors had been snoozing at the wheel before Washington issued 
a clarion call. As previously noted, the education issue was prominently in the agenda of the NGA 
in that same year, and a group of governors, business leaders, and other civic leaders, organized 
by The Education Commission of the States, issued its own call to action in 1983, titled Action for 
Excellence: A Comprehensive Plan to Improve Our Nation’s Schools. When James Hunt spoke 
to the NGA in August, he noted that he and others had been working on that document for six 
months, placing the origin before the April release of A Nation at Risk. 

Rather than just getting under way, Furhman observed that 1986 was, for education, “a year 
in which states and school systems consolidated and assessed changes and reforms of previous 
years.” 


article that simply made the observation that schools were not under the mayor’s control. The 1987 reference was to the 
role of mayors in schools in Chile. 
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Starting in the early 1980s, most states embarked on ambitious attempts to improve the quality 
of public education, Governors convened task forces consisting of business, citizen, and educational 
leaders; some 250 or more task forces were convened to address education reform issues, Legislatures 
passed omnibus bills that incorporated many of the task force recommendations and bundled dozens of 
separate programs together. State boards and departments of education often embarked simultaneously 
on reform programs of their own. (Fuhrman, 1987, p. 131) 


By the late 1980s some of the rhetoric and ideas were beginning to be manifest in stronger 
gubernatorial interventions. One of the 1987 articles using the “education governor” label focused 
on Thomas Kean, a moderate Republican who served as the governor of New Jersey from 1982 
to 1990 (Mooney, 1987). By 1988, frustrated with the lack of sufficient progress, Kean had 
pushed New Jersey to become the first state to initiate takeovers of failing urban schools systems. 
Jersey City, the state’s second largest district, was taken over in 1989. Paterson, the state’s third 
largest, was taken over in 1991. Other states followed New Jersey’s lead. By 2001, 24 states had 
legislation on the books that would allow them to take over failing districts (Wong & Shen, 2001). 

The growing involvement of governors occurred both within their own states and in 
their collective role on the national political scene. Chronicling the changes in the constel- 
lation of interest groups with access to the Congressional deliberations that culminated in 
the No Child Left Behind Act (NCLB), Debray-Pelot includes the NGA among those on 
the ascendance, whereas interest groups representing components of the education sphere— 
the unions, the Council of Chief State School Officers, the American Association of School 
Administrators, the National School Boards Association—were frozen out (Debray-Pelot, 
2007). 

Although in the 1980s the governors were mostly playing a role as agenda setters, using 
their bully pulpit to push the issue of school reform at home and nationally, recently there 
are signs that they are trying to actively wrest more control over their own bureaucracies in a 
shift from agenda-setting to implementation. Governors have direct responsibility for appoint- 
ing state school chiefs in only slightly more than one fourths of all states. In another fourth, 
the school chiefs are separately elected, and in about half of the states they are appointed by 
state boards of education.® “Convinced of the connection between the quality of their schools 
and the future of their states—not to mention their own political reputations,” Education Week 
reported in January 2008, “some governors are seeking a bigger role in shaping education pol- 
icy by grabbing for more control over their state schools chiefs.” Examples included Maryland 
first-term governor Martin O’Malley, who for a time was demanding that Nancy Grasmick, 
the longtime appointed chief, step down so he could work with his own superintendent. “In 
Massachusetts, another first-termer, Gov. Deval Patrick, wants to create a powerful education 
secretary’s job in his Cabinet. And in Indiana and South Carolina, where the chiefs are elected, 
debates swirl over whether the governor should instead pick the superintendent” (McNeil, 2008, 


pre 


Even when they do not appoint them directly, governors can have a strong indirect role in the selection of school 
chiefs when the state board members are themselves selected by the governor or, even when the board is separately 
elected, when the governor’s political base is so strong that he or she exercises a credible threat to unseat board members 
who are too independent in their ways. 
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Presidential Politics 


At the level of national politics, also, there were foreshadowings of a growing presidential role 
before the contemporary area of school reform. Eisenhower (albeit reluctantly) and Kennedy 
both oversaw striking increases in the federal role as it applied to desegregation. In the spring 
of 2007, the House voted to rename the U.S. Education Department’s headquarters after Lyndon 
Baines Johnson. Congressman Gene Green (D-Houston), in offering the measure, proclaimed that 
“T yndon Johnson’s first priority in life was education, and he was the first education president.” 
(p. 4) Backing up that claim was the fact that Johnson had signed more than 60 education bills 
during his presidency, including the path breaking Elementary and Secondary Education Act and 
the creation of Head Start (Mittelstadt, 2007). 

Nonetheless it fell to George H. W. Bush to self-consciously lay claim to the “education 
president” label. At the time it seemed ironic that a Republican would take this stance—especially 
a vice president in an administration that had aggressively sought to get the national government 
out of the education business. To some extent, it may have been a tactical move, a deliberate effort 
on the part of the Bush campaign to distinguish him from Reagan as a candidate with a proactive 
domestic and social agenda. Moreover, by Reagan’s second term, conservative Republicans had 
discovered, in the form of school choice, a federal education platform that they could rally behind. 

In office, Bush, Sr. combined high visibility pronouncements on education with a generally 
lackluster legislative record in terms of actually getting things done. In September 1989, Bush 
assembled a group of governors in Charlottesville, Virgina, for an “education summit” intended to 
launch a partnership to establish national education goals. By working so directly with the nation’s 
governors, he was able to combine a cheerleader role with traditional Republican deference to 
states rights. 

Although Lauro Cavazos, Bush’s first secretary of education, confined himself to a minimalist 
agenda, reflecting the belief that the legitimate federal role was “finding and publicizing success 
stories at the state and local level” (Henig, 1994, p. 90), things picked up a bit under Cavazos’s 
successor, Lamar Alexander. Alexander was a former governor of Tennessee. After engaging 
in high profile battles to attract new manufacturing jobs to the state, he had begun to focus 
on education reform in the state, one of the first so-called education governors. Pushing such 
measures as tougher math and science requirements, merit pay for teachers, and mandatory 
kindergarten, “he concluded that Tennesseans would not get better jobs unless they got better 
educations” (Chira, 1991, p. 52). With Alexander taking the lead, the Bush administration began 
to take a more activist stand, and this accelerated even more as his first administration wound to 
a close and he found himself in a tough battle for reelection. 

Bush was not the pro-market ideologue that Reagan had been, and his vision of education policy 
included a role for federal standards, not simply letting parents decide. Bush, nonetheless, was 
being criticized in some quarters for having failed to generate any real momentum on education 
reform, and as the next presidential election began to loom on the agenda, he began talking more 
about education and at the same time began adopting more of the pro-voucher, market-oriented 
language that he had previously downplayed. In January 1992, less than one month before the 
New Hampshire primary, Bush announced a planned “GI Bill for Children.” Labeled “Federal 
Grants for State and Local ‘GI’ Bills for Children,” the proposal sought simultaneously to conjure 
a major governmental initiative and maintain rhetorical allegiance to the notion that the national 
government would be confined to the role of helping states. The legislation would have provided 


MAYORS, GOVERNORS, AND PRESIDENTS 289 


grants to states and localities that offered thousand dollar “scholarships” to low- and middle- 
income families; these scholarships could be used to pay for supplementary educational services 
(such as after-school programs) or applied toward tuition at private secular or religious schools. 

In emphasizing education, George H. W. Bush was responding to a changing set of public 
expectations for presidential leadership in this area. From the 1960 election, pitting Kennedy 
versus Nixon, through the 1984 election between Mondale and Reagan, education had never 
ranked among the top 10 issues named as “most important” to the American public. When Bush 
ran against Dukakis in 1988, education issues ranked eighth out of 26 in the months leading 
up to the election; in 1992, when he ran against Clinton, they ranked fifth (McGuinn, 2006, p. 
149). As Fuhrman observed, “as the 1990s began, it was clear that some resurgence in federal 
leadership would follow, but the shape of a revitalized federal role was not yet apparent.” By 1994, 
she concluded, “‘a more vigorous federal presence in education was evident” (Fuhrman, 1994, 
pp. 83-84). 

Goals 2000, signed into law in March 1994, provided the basic framework of the Clinton 
approach, with its emphasis on systemic reform, national standards, development of voluntary 
standards and assessments, and teacher quality and professionalism. Somewhat ironically in 
view of the fact that Democrats are typically characterized as favoring big government, Clinton 
emphasized a national rather than federal approach—working with organizations in civil society 
like professional associations and framing standards as aspirational goals not mandatory check 
points. In a classic “Nixon to China” move, it took a Republican president to really up the ante 
in the federal governmental role. 

Although George W. Bush did not as self-consciously pursue the “education president” hon- 
orific as had his father, some argue that his NCLB achievement marked him as being the president 
who most dramatically expanded the reach of the White House. Those who see Bush in these 
terms note the precedent-setting aspects of NCLB. Although the law sidestepped direct preemp- 
tion of state prerogatives to set curriculum and define standards, it nonetheless used the leverage 
of federal funds to forcefully steer states toward some areas of learning (especially math and 
reading) over others. Although the legislation owed much to a bipartisan Congressional coalition, 
its implementation was managed out of the White House in ways that many in Congress had not 
anticipated. The Bush Administration put its direct stamp on policy through the tight control it 
exercised over the U.S Department of Education, placing political appointees in key positions 
and creating provisions in the law to promote the administration’s agenda as advocates for char- 
ter schools, choice, and the use of private management organizations as contracting partners to 
reconstitute failing schools and districts. 

But spending on education under the Bush administration never reached the levels some 
believed he had promised during the negotiating phases leading up to NCLB. Some of the articles 
on the “education president” that are included in Figure 1 employ the term specifically to argue 
that Bush failed to live up to that standard. One refers to him as the “almost education president” 
(Oliphant, 2002). This line of criticism raises a question of whether at least some of the visibility 
of education executives might be more symbolic and rhetorical—focused on the agenda-setting 
side of the policy cycle—rather than indicative of a permanent and substantive involvement that 
carries through to implementation and results. 

More generally, it is important to note that every contemporary president had broader ambi- 
tions for shaping educational policy than could be realized given the interests and institutions 
they confronted. Both conservative and liberal presidential initiatives in education have been 
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constrained by what Congress, the Courts, and the reigning fiscal conditions have allowed. De- 
spite his popularity, Reagan’s goals to dismantle the Department of Education and institute a 
broad voucher scheme were thwarted by Congress. If they had succeeded, this success would 
have constituted a huge change in the nation’s educational system, despite the fact that arguably 
they would have reduced the role of future presidents by pushing key decisions from Washington 
to the states and from government to market forces. Both George H. W. Bush and Bill Clinton had 
to contend with strong states rights constituencies, leading them to call more for voluntary than 
mandated changes and favor “national” goals and initiatives, which might be pursued through 
professional organizations and civil society, over “federal” ones in which government would play 
the leading role (Kaestle, 2007). 


Mayoral Control 


Mayoral control of schools in itself is nothing new. Indeed, prior to the Progressive Reform Era 
in the early 20th century, public education most often was housed in an agency reporting to a 
mayor much as would be the case with law enforcement or public works. What is new, however, 
is its reemergence as a favored reform after decades in which a rival vision held sway. 

During the Progressive Era many communities moved to remove oversight of schools from 
the mayor’s responsibility, in the belief that this would lead to a more professional approach and 
one less responsive to partisan politics and the temptation to use school jobs and school building 
contracts as a form of patronage. The strategy for depoliticizing education was to place school 
governance in the hands of an independently selected school board, often elected off-cycle from 
general elections, and often with a dedicated revenue stream. Some urban districts, like Chicago, 
New Haven, and Jackson, Mississippi, never took schools out of the mayor’s portfolio of control, 
but of the large districts currently under mayoral control most have moved back in that direction 
over the past 15 years. 

Boston’s elected school board’ for years had been embroiled in the city’s turmoil over de- 
segregation and was denigrated by many for perceived petty politics, fiscal mismanagement, and 
general disarray. Although members of the city’s business and municipal reform communities had 
been calling for governance changes for years, mayors Kevin White (1967-1983) and Raymond 
Flynn (1984-1993) were reluctant to step into the breach. As Flynn at one point explained to the 
business community, “Public education is an area that can swallow up the most promising career 
and politicians are counseled at every step to ‘stay away from the schools”’ (as cited in Portz, 
2004, p. 99). Despite his wariness, and despite a 1989 referendum showing the Boston public to 
be divided on the issue, pressure to replace the elected board with one appointed by the mayor 
continued to grow. In 1991, the city council petitioned the state to make the change, the state 
agreed, and a new board appointed by Flynn took office in January 1992 (Portz, 2004). 

Chicago is typically counted as the second large city to shift to mayoral control in the modern 
era, although this is misleading in some senses. Although several governance reforms altered the 
specific details, Chicago mayors have had the authority to appoint the school board going back 
at least until 1870 (Shipps, 2006). For much of that time this formal power was used gingerly, at 


In Boston, the body was known as the school “committee.” 
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least in terms of involvement with the substance of education.’ When the state legislature passed 
a 1995 law restructuring school system governance in the city, three factors combined to turn it 
into more of a watershed event. First, the 1995 law came on the heels of—and in some ways as 
a retraction of—a path breaking 1988 reform that had radically decentralized some elements of 
school decision-making to local school councils. Second, among the formal powers assigned to 
the mayor in 1995 were expansions of discretion in areas of budgeting, contracting, and personnel. 
Third, after an initially cautious start, Daley began to aggressively and very publicly step into 
the leadership role, wading into more substantive issues of school policy with a confidence and 
authority that his predecessors, including his famous father, had been reluctant to demonstrate 
(Shipps, 2004). Perhaps because Thomas Menino, Flynn’s successor in Boston, took a generally 
low visibility role in exercising his role, Daley and Chicago probably had more of an impact in 
bringing mayoral control to national attention. 

Subsequent transitions to mayoral control varied in their particulars. In some cases, for ex- 
ample, sitting mayors were eager aspirants for authority, whereas in others they more reluctantly 
took on the role. What was common across the cases was a political alignment that put the local 
business community and state officials on one side, as proponents of the change, with the local 
African American population often reticent or openly resistant (Henig & Rich, 2004). 

Mayors in the forefront of the school takeover movement relied heavily on the local business 
community for the political and financial support to initiate the change and help them make a 
quick impact. Some added their own vision and political muscle, but in other cases the mayors 
rode a political wave that they had little role in creating. To corporate leaders, a structure 
in which there is integrative management culminating in a chief executive officer is familiar. 
Corporate leaders also feel culturally more comfortable and more confident of capturing the ear 
of today’s “modern mayor’ than they do with school board members, many of whom are steeped 
in the language and ideas of the education community and many of whom are accountable to 
ward-based constituencies whose views business leaders consider parochial and particularistic 
(Kirst & Bulkley, 2000). For various reasons, some previously discussed, states had been getting 
increasingly involved in the challenge of urban school reform, but efforts to take direct control of 
local districts —as New Jersey had pioneered in Paterson, Elizabeth, and Newark, and as Congress 
attempted when it imposed a Financial Control Board on the District of Columbia—had revealed 
that running a district from the state capital in the aftermath of a hostile takeover was extremely 
challenging on political and practical terms. Mayoral control was appealing on this front as a way 
to empower a new partner, one with a claim to an electoral mandate and with the local knowledge 
and administrative capacity to get things done. 

This alliance of business leaders, state legislatures, and reform-minded mayors was regarded 
skeptically by many local residents, who saw the movement as an assault on traditional and 
familiar institutions. Often, this divergence in perspective has had a racial overlay. The African 
American population in particular has identified with the local public schools system as a source 
of jobs, of upward mobility, and community development, and where mayoral control initiatives 
have been put to a vote the racial cleavages have been pronounced (Henig, 2004). Although 
race-based loyalties play a role, some of the resistance to mayoral control stems from a more 
generic reaction to a systemic change that has the potential to reallocate influence. Unions, 


8Relative disinterest in the mechanics of schooling did not prevent some of the city’s mayors from aggressively using 
the schools as a source of patronage and power. 
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community-based organizations, and parent groups that had channels of access to elected school 
boards are concerned that mayoral control leaves them more marginalized (Chambers, 2006; 
Hemphill, 2009). 


RIVAL EXPLANATIONS: MOVING BEYOND LEVEL-SPECIFIC 
ACCOUNTS 


What accounts for these apparent trends and the fact that they seem to have emerged at more 
or less the same time? To date, theorizing and research around this question has been relatively 
spare and almost exclusively level-specific. By “level-specific,” I mean that the literature on the 
national, state, and local level governance changes has comprised three separate streams with 
little cross-fertilization. That pattern is not unique to this question of the role of executives. To 
some extent it reflects lines of organization within political commentary, the media, and political 
science, and that mimic the layer effect of our federal system. Each of these arenas tends to have 
specialists who focus on the national politics playing out in Washington, DC. The structure of 
political commentary and media overlaps to a considerable degree. The so-called talking heads 
who offer commentary in national publications and on weekend news shows typically focus on the 
inside baseball stories that involve the White House, Congress, Supreme Court, major agencies, 
and national parties. There are also political analysts with expertise on local politics, but these are 
a different cast of characters who tend to have specialized local knowledge; they have sources in 
City Hall and the school district offices, can call the superintendent or local union leader, and are 
experts in the personalities and political history of their local beat. These local specialists follow 
events at the state level when they bear upon local issues, but rarely in-depth or on their own 
terms. 

This bifurcation of expertise (local vs. national with the state as a bit of a doughnut hole) 
is also found in the media, and the relationship is more than coincidental. Newsmagazines and 
network television have national audiences; they focus on events in this or that city or state only 
when they have broad relevance and appeal. Most newspapers and radio stations are localized 
in audience and coverage where stories about the statehouse and legislature are occasional and 
idiosyncratic, with a special emphasis on the local angle. 

Political science, public administration, and public policy, as arenas of academic specialization, 
are just as fragmented across the layers of federalism. Some political scientists specialize in 
American national politics, usually focusing either on public opinion and voting behavior or 
national institutions like Congress and the Supreme Court. Although it is not uncommon for 
there to be courses offered on “state and local politics,’ these are often grouped together as a 
matter of convenience; in general, specialists in urban politics tend to be different from those who 
specialize in state politics. To illustrate the factionalization within the layer cake of federalism, in 
the organized sections of the American Political Science Association, as of November 2004, only 
13 of the 417 members of the Urban Politics section and of the 643 members of the Elections, 
Public Opinion, and Voting section were cross-listed.? 


°Data, as of November 10, 2004, are available at http://www.apsanet.org/section302.cfm. This is meant as an illustra- 
tion, not an empirical verification of the intellectual fragmentation thesis. Scholars, of course, may borrow ideas across 
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We need to step outside this compartmentalized analysis to get some perspective on whether we 
are experiencing three distinct phenomena or something broader and perhaps of greater import. 
As a first step, I offer five rival hypotheses to account for the observed patterns of increased public 
involvement in education by presidents, governors and mayors. Testing them separately is beyond 
the scope of this paper, although I make some general comments on their relative credibility. 


Situational and Opportunistic 


One possibility is that the explanation for growing executive involvement in education lies in 
events that are specific to particular levels of government, particular jurisdictions, and transitory 
external events. Many of the existing, more journalistic, accounts seem to support this possibility. 
Narratives about the emergence of mayoral control in places like Boston, Chicago, DC, and New 
York City, for instance, suggest that specific elements of local political history are important 
in explaining dissatisfaction with the traditional elected school board structure and readiness to 
try something new—the role of the board in the bitter struggles over desegregation in Boston; 
the long-term involvement of the business sector in Chicago and the short-term experience with 
a prior reform measure that, in some eyes, took decentralization too far; years of intramural 
squabbling on the elected board in DC combined with a history of takeover by Congress. 
Personalities have almost certainly influenced the timing of shifts in the executive role. No 
one believes that Congress would have permitted mayoral control of schools in DC while Marion 
Barry was the chief executive; the New York State legislature refused to enact mayoral control 
when then mayor Giuliani was the one calling for power; in Cleveland and Detroit, similarly, state 
legislatures were eager to empower the mayors in very large measure because of comfort levels 
with Michael White and Dennis Archer, respectively. James Hunt, who played a role in prodding 
the governors in addressing education on a national skill, was by most accounts an exceptional 
leader, highly respected by his peers; that his personal interest in education transcended short-term 
tactical calculations is reflected in the fact that he continued to devote time to this issue well after 
he left office. There is a situational element, too, to George W. Bush’s decision to feature education 
reform in his 2000 campaign and to the alignment of factors that helped bring about NCLB’s 
bipartisan support. Nothing in Bush’s early career suggested a deep interest in or knowledge 
about education, but gains in Texas test scores while he was governor gave him an opportunity to 
present this as a domestic policy area in which he had a track record.!° Once he made NCLB his 
number one priority, Republicans in Congress fell in line, even those who would have resisted 
this expansion of the national government’s power, out of party loyalty and a belief that it was 
strategically important to give their president a victory. Senator Edward Kennedy also figures into 
this story. Kennedy might not have been able to block NCLB, but he certainly could have made 
its passage more difficult. Instead, Kennedy calculated that cooperation would keep him in the 
negotiation process. He could use the legislation as a way to extract a larger funding commitment 
for education than would be forthcoming from the Republicans on their own initiative. 


the boundaries of organizational subdivisions even if they do share membership. A more careful test would call for a 


content and citation analysis. 
10For an account of media coverage of the so-called Texas Miracle, as well as analysis that calls its legitimacy into 


question, see Haney (2000). 
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According to the situational perspective, the explanation for expanding executive involvement 
has to do with transient opportunities and the calculated responses to them by key actors. It 
is relatively easy to construct a narrative around the emergence of every education president, 
governor, and mayor that conforms with this template. At the same time, one of the weaknesses 
of this hypothesis is the very fact that it is so elastic as to be virtually unfalsifiable. Moreover, 
the situational hypothesis is on shaky grounds in accounting for what seems to be a systematic 
increase in the emergence of the phenomenon and its contagion across the levels of government. 
The most credible form of the situational hypothesis would seem to require the corollary that 
the appearance of simultaneous and systemic change reflects selective perception and framing; 
scattered and idiosyncratic examples being packaged as if they represent a phenomenon more 
broadly based than the objective assessment would uphold. If opportunities and incentives really 
are shifting broadly in a way that leads rational actors to settle on elected general purpose 
executives as the favored focal point for educational reform, our quest for explanations needs to 
look toward these more systemic and structural factors and not focus on tactical decisions made 
in response. 


Intergovernmental Imitation and Diffusion 


The spread and sequencing of the move to a larger role for presidents, governors, and mayors 
could be a consequence of intergovernmental imitation and diffusion. Scholars of policy diffusion 
have long recognized that states often look to their neighbors for innovative policies (Gray, 1973; 
Walker, 1969). More recent research shows that states also learn to copy one another via other 
channels, such as professional associations (Balla, 2001) and policy entrepreneurs (Mintrom, 
2000). In the case of mayoral control, later adoptors very self-consciously copied from Boston, 
Chicago, and New York City. New York’s Michael Bloomberg was particularly cooperative 
in sharing his insights and experience with others, like Villaraigosa (Los Angeles) and Fenty 
(Washington, DC; Medina & Herszenhorn, 2007). 

Historical accounts support the notion that a similar diffusion process might operate across 
as well as within levels of government. Governors, seemingly the first among the elected ex- 
ecutives to systematically adopt the “education” label, helped set the stage for the education 
presidency. This was the case both collectively, as expressed through the Education Commis- 
sion of the States and the National Governors Association, and via individual governors who 
carried their education governor mantel onto the national stage, such as Lamas Alexander 
and Richard Riley (as secretary of education under George H. W. Bush and Bill Clinton, re- 
spectively) and Bill Clinton (as president). The diffusion may also travel from state to local 
level. Roy Romer, for example, after serving in the legislature and as a three-term governor of 
Colorado went on to become CEO of the Los Angeles Unified School District from 2000 to 
2007. 

The imitation and diffusion hypothesis accounts for the systemic pattern of adoption better than 
the situational hypothesis. Unless elected executives and those who have the power to influence 
governance reform are unthinking mimics, though, it leaves unexplained the question of why 
it is this specific reform strategy that is being spread. In light of the long history of education 
reforms that rapidly diffuse and just as rapidly fade (Hess, 1998; Tyack & Cuban, 1995), it is 
not inconceivable that handing power to elected executives is just another in a long list of what 
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Farkas (1992) labeled “reform du jour.” If the phenomenon proves to be a sustainable change, 
however, we may need to look for even more structural explanations. 


Business Interests as Multilevel Actors 


It is possible that the growing centrality of presidents, governors, and mayors reflects less their own 
power than the increasing interest of the corporate sector in education. The reform coalition that 
culminated in A Nation at Risk, the Education Summit at Charlottesville, the general pressure 
for standards and accountability, and the turn toward mayors at the local level owed much to 
the leadership of the business community. In the battle among interest groups at all levels of 
American politics, business has always had a “privileged position” (Lindblom, 1977). At the 
state and local levels, businesses for many decades showed generally less interest in schools 
than taxes, economic development, and physical infrastructure. During the 1980s, however, some 
major corporate sector leaders began to focus more on investment in human capital as the way 
to compete in the emerging global economy. This showed in the involvement of the Business 
Roundtable and Committee on Economic Development in helping to articulate the nation’s new 
school reform agenda. 

During the Progressive Era, business interests often considered mayors and governors to be 
driven by partisan and patronage-based politics and favored governance structures designed to 
empower bureaucracies buffered from mayoral influence. Over time, however, both education 
bureaucracies and school boards became more remote from business, more political in style, 
less open to market ideas. The growing alienation between business and elected boards inter- 
sected, during the 1980s and since, with the emergence on the political scene of a “new style” of 
pragmatic, management-oriented governors and mayors (Kirst & Bulkley, 2000; Sabato, 1978). 
In some cases business was part of the electoral coalition that brought these pragmatists into 
office, but even where that was not the case, business often subsequently discovered that gov- 
ernors and mayors were more likely to speak their language and share their policy assumptions 
than legislators accustomed to selling directly to the average voter or school district leaders 
socialized to protect the educational status quo. The combination, then, of growing business in- 
terest in education and growing business comfort with its access to and influence on mayors and 
governors, may have combined to make business a major carrier of the strong executive branch 
theme. 


More Muscular Executives 


Also having more long-term implications than an explanation based on ad hoc opportunities 
or simple diffusion and imitation is the hypothesis that the growing prominence of presidents, 
governors, and mayors represents a strengthening of the executive branch across most policy 
issues. Changes in the tasks of governance, in popular conceptions of governance, and the 
impact of media all may be giving elected executives a more powerful platform for engagement 
and making it more feasible for them to expand their reach into areas they previously left to 
legislatures, courts, or their agency heads. 
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As government grows—in personnel; in professionalism; in scope of responsibility; in regu- 
latory reach, management and administration—the challenging task of riding herd on the gov- 
ernmental apparatus becomes more important in actuality and in people’s minds. That, arguably, 
creates an environment in which the executive branch stands to gain in stature relative to the 
legislature and judiciary. Not just in the United States, but throughout the Western world, analysts 
have commented on the rise of the “regulatory state,’ a governance regime in which rules, over- 
sight, and public contracting to some extent displace direct government provision, legislation, 
taxing and spending as the mechanisms through which governments operate (Majone, 1997). 
New data systems that give agencies advantages also may add to the musculature of the executive 
branch. 

Some of these changes speak more to the potency of the executive branch than to the chief 
executive per se. Indeed, in some ways the forces at play may make it more difficult for a pres- 
ident, governor, or mayor, as principal, to steer the bureaucracy as his or her putative agent. 
Relative to the other branches, though, the executive gains extra clout. And although analysts 
of the regulatory state suggest that the emphasis on rules can also give increased power to the 
judiciary, recent battles in the United States over executive privilege and its role in responding 
to terrorism suggest at least the potential for a strong executive to stonewall a probing judi- 
ciary. The growing importance of the media in simplifying issue and policy framing for the 
public also gives a president, governor, or mayor opportunities to gain leverage, through their 
greater ability to commandeer the “bully pulpit,” over faceless bureaucracies and multiheaded 
legislatures. 

From this perspective, the symbiotic relationship between business and executives might be 
seen in a different light. Rather than business interests helping to create stronger executives to 
improve their access and empower a sympathetic ally, the causal arrow may run the other way. 
Increasingly powerful mayors, governors, and presidents may have cultivated business as part of 
their electoral and governance regimes, steering them to support their agendas to at least the same 
degree as the other way around.!! 


The End of Educational Exceptionalism and the Incorporation of Education Policy 
into General Purpose Government 


Finally, it is possible that something else is going on. The growing role for elected executives 
may not be the real or whole story. Rather, what has been characterized in those terms may be a 
consequence of the breakdown of the traditional separation between education as a form of special 
government—a functional arena with its own decision-making structures—and its re-absorption 
into general purpose politics and governance. 

The historic buffering of education in its own semiautonomous governance structures has 
always been somewhat anomalous. Proponents of public education favored this separation in part 
out of the belief that investment in schooling would erode if forced to compete on even terms 
with pressures to spend on other areas, to keep taxes low, and to treat school jobs as patronage. 
Within a buffered arena, those who cared deeply about education would be able to protect it from 


'IThis is the argument that Robert Dahl famously developed to counter stratificationists’ claims that business elites 
were running American cities (Dahl, 1961). 
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an indifferent or distractible majority. The notion that education was special also received support 
from the arguments of those sensitive to the critical role of education in socializing children, 
and the consequent risks if a repressive political regime were to use schooling to manufacture a 
supportive or docile citizenry. Regardless of whether those notions were correct at one time, or 
correct still, it appears credible that they have lost resiliency in the contemporary environment. 
To the extent that the need to compete globally makes the members of the general public more 
cognizant of the importance of public education even when they do not have children in the 
schools, it is credible that proponents of educational investment and reform can make their case 
in the broader arena of general purpose politics. To the extent that the traditional core supporters 
of education are structurally weaker—because of the declining access and clout of the teachers 
unions, and to a student enrollment with higher proportions of minorities who are regarded less 
favorably as policy beneficiaries and less able, politically, to defend their interests—proponents 
of public education may have no real choice but to frame their arguments and build their coalitions 
with an eye to competing in broader political arenas. 

If this is what is going on, the broad shifts we are witnessing come at the expense of ed- 
ucation specific bodies—local and state boards of education—but, significantly, not necessar- 
ily at the expense of either the legislatures or courts. Anecdotally, there is evidence that the 
movement toward more involvement by presidents, governors, and mayors may, indeed, cor- 
respond with a growing level of involvement from these other institutions of general purpose 
government. !? 


CONCLUSION 


And something is happening here 
But you don’t know what it is 

Do you, Mister Jones? 

— Bob Dylan (1965) 


Rather than a series of localized and idiosyncratic reforms, Charles Kerchner, William Boyd, 
and others (Boyd, Kerchner, & Blyth, 2008; Kerchner, Menefee-Libey, Mulfinger, & Clayton, 
2008) have suggested that the dynamics of big-city education politics reveals “a relentless march 
toward new institutional forms” (Kerchner et al., 2008, p. 2). They began their analyses with 
a focus on schools and school systems and worked upward and outward to this more systemic 
perspective. In this article, I started on the outside, with attention to elected chief executives. 
Our analyses converge on the point of institutional change. Our causal emphases differ, perhaps 
because of the differing starting point. They present the dynamics primarily in terms of forces 
operating within the educational decision-making arena. The story unfolding here suggests that 
key political levers may originate outside of that arena. 


12See Labb and Haynes (2008) for insight into the ways in which mayoral control in DC may be ratcheting up the 
city council’s involvement. Portz and Schwartz (2009) indicated that “increasingly, community activists are turning to 
the Boston City Council as a venue to raise concerns and grievances” (p. 103). Historically, as governors took the lead in 
asserting a stronger state role in education, state courts became the targets of equity and adequacy suits that brought them 
more clearly into the fray. Although George W. Bush has been portrayed as the force initiating NCLB, the White House 
actually offered only a sketchy proposal, leaving it to Congress to flesh out the specific details. 
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Although the visible face of the march toward new institutional forms has been a growing 
role for presidents, governors, and mayors, understanding of why this is, and what may be 
the consequences, has been hampered by the tendency to provide situational and particularistic 
accounts tied to particular jurisdictions and particular levels of governance. I have suggested here 
that there are rival, and more structural, explanations. Each has different implications about what 
is driving the change and, by extension, how it is likely to develop and, perhaps, the consequences 
we can anticipate. 

My own view of this point is that the unifying feature of executive prominence may be 
best understood in terms of the end of educational exceptionalism, but I offer this less as a 
conclusion than as a hypothesis that is open to empirical probing. To take this on seriously, 
means that researchers may need to venture beyond the normal and comfortable confines of 
specializations that reflect the old layer cake image of federalism. It also means that those of us 
who study the politics of education may need to think more about studying politics and education. 
That does not mean overlooking what is distinct about the institutions, interests, and ideas that 
characterize conflict and coalition building around the issues of schooling, but it does mean more 
self-consciously attending to the ways in which these interact with—and can be affected by—the 
institutions, interests, and ideas that animate general purpose politics and governance. 


REFERENCES 


Balla, S. J. (2001). Interstate professional associations and the diffusion of policy innovations. American Politics Research, 
29, 221-245. 

Bluestone, B., & Harrison, B. (1982). The deindustrialization of America. New York: Basic Books. 

Boyd, W. L., Kerchner, C. T., & Blyth, M. (Eds.). (2008). The transformation of great American school districts: How 
big cities are reshaping public education. Cambridge MA: Harvard Education Press. 

Caldwell, P. (1985, February 6). Governors: No longer simply patrons, they are policy chiefs. Education Week, pp. 1; 34. 

Chambers, S. (2006). Mayors and schools: Minority voices and democratic tensions in urban education. Philadelphia: 
Temple University Press. 

Chira, S. (1991, November 24). Lamar Alexander’s self-help course. New York Times Magazine, p. 52. 

Dahl, R. (1961). Who governs? Democracy and power in an American city. New Haven, CT: Yale University Press. 

Debray-Pelot, E. (2007). Dismantling education’s “iron triangle’: Institutional relationships in the formation of federal 
education policy between 1998 and 2001. In C. F. Kaestle & A. E. Lodewick (Eds.), To educate a nation: Federal 
and national strategies of school reform (pp. 64-90). Lawrence: University Press of Kansas. 

Dylan, B. (1965). Ballad of a thin man. On Highway 61 revisited [Record]. New York: Columbia Records. 

Farkas, S. (1992). Educational reform: The players and the politics. New York: The Public Agenda Foundation. 

Fuhrman, S. F. (1987). Education policy: A new context for governance. Publius, 17(3), 131-143. 

Fuhrman, S. H. (1994). Clinton’s Education policy and intergovernmental relations in the 1990s. Publius, 24(3), 83-97. 

Gray, V. (1973). Innovation in the states: A diffusion study. American Political Science Review, 67, 1174-1185. 

Haney, W. (2000). The myth of the Texas miracle in education. Education Policy Analysis Archives, 81(41). Available 
from http://epaa.asu.edu/epaa/v8n4 1/ 

Hemphill, C. (2009). Parent power and mayoral control: Parent and community involvement in New York City schools. 
In J. P. Viteritti (Ed.), When mayors take charge: School governance in the city (pp. 187-205). Washington, DC: 
Brookings Institution Press. 

Henig, J. R. (1994). Rethinking school choice: Limits of the market metaphor. Princeton, NJ: Princeton University Press. 

Henig, J. R. (2004). Washington DC: Race, issue definition, and school board restructuring. In J. R. Henig & W. C. Rich 
(Eds.), Mayors in the middle: Politics, race, and mayoral control of urban schools (pp. 191-220). Princeton, NJ: 
Princeton University Press. 

Henig, J. R., & Rich, W.C. (Eds.). (2004). Mayors in the middle: Politics, race, and mayoral control of urban schools. 
Princeton, NJ: Princeton University Press. 

Hess, F.M. (1998). Spinning wheels: The politics of urban school reform. Washington, DC: Brookings Institution. 


MAYORS, GOVERNORS, AND PRESIDENTS 299 


Hess, F. M., & Finn, C. E., Jr. (2007). Introduction. In F. M. Hess & C. E. Finn, Jr. (Eds.), No remedy left behind (pp. 
1-14). Washington, DC: AEI Press. 

Hess, F. M., & McGuinn, P. J. (2002). Seeking the mantle of “opportunity”: Presidential politics and the educational 
metaphor, 1964-2000. Educational Policy, 16(1), 72-95. 

Hoffman, D. (1988, February 28). Bush returns to pre-lowa stump speech; Attacks on Dole dropped in South. The 
Washington Post, p. AO9. 

Hunt, J. (1983, July-August). Discussion of education for economic growth and consideration of proposed policy position. 
In Proceedings of the National Governors’ Association Annual Meeting 1983 (pp. 6-8). Washington, DC: National 
Governors’ Association. Available from http://www.nga.org/Files/pdf/1983NGAAnnualMeeting.pdf 

Kaestle, C. F. (2007). Federal education policy and the changing national polity for education, 1957-2007. In C. F. Kaestle 
& A. E. Lodewick (Eds.), To educate a nation (pp. 17-40). Lawrence: University Press of Kansas. 

Kerchner, C. T., Menefee-Libey, D. J., Mulfinger, L. S., & Clayton, S. E. (2008). Learning from L.A.: Institutional change 
in American public education. Cambridge, MA: Harvard Education Press. 

Kirst, M., & Bulkley, K. (2000). “New, improved” mayors take over city schools. Phi Delta Kappan, 80, 538-546. 

Krotseng, M. V. (1987, November). The “education governor”: Political packaging or public policy? Paper presented at 
the meeting of the Association for the Study of Higher Education, Baltimore. 

Labb, T., & Haynes, V. D. (2008, February 23). Rhee has ambitious plans but still no school budget. Washington Post, p. 
B10. 

Lindblom, C. (1977). Politics and markets: The world’s political-economic systems. New York: Basic Books. 

Majone, G. (1997). From the positive to the regulatory state: Causes and consequences of changes in the mode of 
governance. Journal of Public Policy, 17(2), 139-167. 

McGuinn, P. (2006). No Child Left Behind and the transformation of federal education policy 1965-2005. Lawrence: 
University Press of Kansas. 

McNeil, M. (2008, January 23). Governors face political hurdles in seeking power to appoint chiefs. Education Week, 
pols 

Medina, J., & Herszenhorn, D. M. (2007, September 3). School year’s about to start, and so, too, is a big change. New 
York Times, p. B1. 

Mintrom, M. (2000). Policy entrepreneurs and school choice. Washington, DC: Georgetown University Press. 

Mittelstadt, M. (2007, March 7). House OKs Texans’ bill honoring LBJ. Houston Chronicle, p. 4. 

Mooney, C. J. (1987). New Jersey’s education governor. The Chronicle of Higher Education, 34(7), A20. 

Oliphant, T. (2002, February 19). Education president reneges on the deal. Boston Globe, p. All. 

Portz, J. (2004). Boston: Agenda setting and school reform in a mayor-centric city. In J. R. Henig & W. C. Rich (Eds.), 
Mayors in the middle: Politics, race, and mayoral control of urban schools (pp. 96-119). Princeton, NJ: Princeton 
University Press. 

Portz, J., & Schwartz, R. (2009). Governing the Boston public schools: Lessons in mayoral control. In J. P. Viteritti (Ed.), 
When mayors take charge: School governance in the city (pp. 91-116). Washington, DC: Brookings Institution Press. 

Sabato, L. J. (1978). Goodbye to Goodtime Charlie: The American governor transformed, 1950-1975. Lexington MA: 
Lexington Books. 

Shipps, D. (2004). Chicago: The national “model” reexamined. In J. R. Henig & W. C. Rich (Eds.), Mayors in the middle: 
Politics, race, and mayoral control of urban schools (pp. 59-95). Princeton, NJ: Princeton University Press. 

Shipps, D. (2006). School reform, corporate style: Chicago I 880-2000. Lawrence: University Press of Kansas. 

Tyack, D., & Cuban, L. (1995). Tinkering toward Utopia: A century of public school reform. Cambridge, MA: Harvard 
University Press. 

Vinovskis, M. A. (2008). Gubernatorial leadership and American K-12 educational reform. In E. Sribnick (Ed.), A legacy 
of innovation: Governors and public policy (pp. 185-203). Philadelphia: University of Pennsylvania Press. 

Walker, J. (1969). The diffusion of innovations among the American states. American Political Science Review, 63, 
880-899. 

Wirt, F. M., & Kirst, M. W. (2001). The political dynamics of American education (2nd ed.). Richmond, CA: McCutchan. 

Wong, K. K., & Shen, F. X. (2001, August-September). Does school district takeover work? Assessing the effectiveness of 
city and state takeover as a reform strategy. Paper prepared for the annual meeting of the American Political Science 
Association, San Francisco. 

Wong, K. K., Shen, F. X., Anagnostopolous, D., & Rutledge, S. (2007). The education mayor: Improving America’s 
schools. Washington, DC: Georgetown University Press. 


Copyright © Taylor & Francis Group, LLC 
ISSN: 0161-956X print / 1532-7930 online 
DOI: 10.1080/01619560902973480 


: 200 8 
PEABODY JOURNAL OF EDUCATION, 84: 300-317, 2009 g Routledge 
8 Taylor & Francis Group 


Big City Superintendent as Powerful CEO: Paul Vallas 
in Philadelphia 


Elizabeth Useem 


Research for Action 


This case study of the leadership of Paul Vallas as CEO of the Philadelphia city schools from 2002— 
2007 demonstrates that No Child Left Behind has enhanced the formal authority of big city district 
chiefs even though they are constrained by its accountability mandates and by the pressures from a 
growing number of influential stakeholder groups. In Vallas’s case, he was also able to take advantage 
of powers conferred on him and the district’s School Reform Commission by the state’s takeover 
of the district. District leaders such as Vallas who augment such positional power with their own 
persuasive brand of leadership are particularly well suited for the task of urban district improvement. 


The 2002 federal No Child Left Behind Act (NCLB) and related state legislation have enhanced 
the formal authority of school superintendents in the United States. This is particularly true of 
leaders in urban districts who are charged with shaping a new course for their districts and crafting 
remedies for underperforming schools. The fact that the superintendent’s title has been changed to 
“CEO” or “Chancellor” in some big-city districts signals that their position is intended to be more 
powerful. At the same time, accountability frameworks and other regulations specified place new 
limits on the freedom of district leaders to run their districts as they see fit. Most notably, they 
are required to design and execute interventions that raise student achievement at a rapid clip. 
Paradoxically, then, the “positional power” of superintendents—that is, the authority inherent in 
their office—has become both more robust and more constrained over the last decade. 

A previous body of research has shown that superintendents’ direct power became increasingly 
limited beginning in the 1960s as federal and state roles in education expanded and as organized 
interest groups—teachers’ unions, civil rights organizations, parents of special education students 
and English language learners, among others—successfully pressed their respective cases for 
legislation and regulation (Cuban, 1976; Tyack & Hansot, 1982). Since the 1990s, charter school 
laws at the state level and pressures from private foundations and other non-profits have also 
encroached on superintendents’ zone of control. Superintendents must be skilled at negotiating 
and collaborating with these and other stakeholders and persuading an even wider range of groups, 
including teachers and administrators, to support their policies and initiatives (J. Collins, 2005; 
Cuban, 1988; Cuban & Usdan, 2003; Hess, 1999; Johnson, 1996). 


Correspondence should be sent to Elizabeth Useem, 352 Woodley Road, Merion Station, PA 19066. E-mail: 
buseem @researchforaction.org 
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Even as aspects of superintendents’ positional power have been whittled away by the emer- 
gence of new players and new laws, other aspects of their formal power—notably the ability 
to undertake bold district improvement strategies and to intervene in failing schools in a whole 
variety of ways—have been restored and even expanded by NCLB and other state laws. These 
powers, rooted in the national standards-based reform movement of the 1990s, make it easier for 
district leaders to move from “making an effort” to “getting results.” Leaders in urban districts 
with large numbers of low-performing schools are indeed drawing on these new powers and on 
the culture of standards-based reform to bring about change in their districts (Childress, Elmore, 
Grossman, & Johnson, 2007). In Philadelphia, the site of the case study that follows, the state’s 
takeover of the school system in 2001 further expanded the zone of authority of the district’s 
governing board and the system’s chief. 

This case study looks at the experience of Paul Vallas during his years as Chief Executive 
Officer (CEO) of the School District of Philadelphia (2002-2007) as he navigated the shifting 
cross currents of educational governance in the new standards-based environment. Vallas’s record 
in Philadelphia demonstrates that urban school superintendents continue to have considerable 
power over what happens in their districts, in some ways even more than they had before the 
NCLB era began. As before, their position in the educational hierarchy enables them to articulate 
a vision, set priorities, draw up strategic plans and budgets, initiate a program of instructional 
improvement, design managerial changes, control the information that goes to their governing 
boards and mayors, recommend contractors and personnel, and recommend school closures or 
new school development (Hess, 1999). But Vallas, like district chiefs in other large urban districts, 
had formal powers above and beyond that of district leaders in the recent past (Boyd, Christman, 
& Useem, 2008; Useem, Christman, & Boyd, 2006). 

Vallas’s experience shows that even when other actors assume significant formal roles in school 
governance, a bold and persuasive superintendent who is able to take advantage of enhanced 
formal authority can implement a significant improvement effort in a district. The notion that 
urban superintendents are “whipsawed from within” by their boards, unions, and central office 
staff and “under assault from the outside” (Fuller et al., 2003), although still true, fails to take 
account of the increasing number of large, stressed districts that are run by leaders whose job 
descriptions contain expanded powers. In some notable cases, including Vallas, they complement 
this formal authority with their own influential personal leadership style. 


DATA AND METHODS 


The data that inform this case study come from multiple sources. A collaborating group of 
scholars from five institutions, led by Research for Action (RFA) in Philadelphia and including 
this author, has been gathering and analyzing information on Philadelphia’s reform since its 
inception in 2001. The analysis in this chapter draws on RFA’s archive of district documents and 
articles in the print media as well as its field notes of nearly 100 meetings of the School Reform 


'This research and public awareness project, Learning from Philadelphia’s School Reform, was a multiyear, multi- 
pronged research effort. RFA has published reports on governance, civic engagement, teacher quality, and the impact 
of private educational management organizations on student achievement. See http://www.researchforaction.org for an 
explanation of the project and links to the reports and tools that explain the reform. 
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Commission, important events and meetings of community groups, and regularly scheduled 
meetings of civic groups. The data also include 13 interviews with key civic leaders and highly 
placed district officials and observers during 2005 and 2006, and in-depth interviews of 50 district 
administrators and 35 local leaders of community, advocacy, media, business, civic, and religious 
organizations between 2002 and 2006. RFA also conducted interviews with one or more locally 
based directors of all seven of the original external school management organizations with whom 
the district had contracts during this period. 


The Philadelphia Context 


Philadelphia has a recent history of hiring strong reform-minded superintendents who stay long 
enough to leave their mark on the system.” Mark Shedd, superintendent of the system from 1967 to 
1971, gained national attention for his efforts to bring change to a notoriously hidebound system 
that had been run for years by a dictatorial superintendent (“Ousting,” 1971; Resnik, 1969). 
Constance Clayton ran the system for 10 years (1983-1993) and succeeded in stabilizing labor 
relations, implementing a district-wide curriculum, establishing reform-support organizations 
in the nonprofit community, and breaking down the districts 22 comprehensive high schools 
into schools-within-schools (Christman & MacPherson, 1996; Fine, 1994). Clayton’s reforms, 
however, did not lead to increases in student test scores. Her control over information on student 
performance would not have been possible today in a world where information on student 
outcomes must be made public. 

Clayton was followed by David Hornbeck (1994-2000) who initiated a sweeping 10-point 
reform plan. Its centerpiece was a focus on improving student achievement, including a system 
of academic standards and accountability, a decentralized system of school “clusters” that put 
more instructional authority in the hands of school leaders, enhanced professional development 
for teachers, and greater support for students and their families. But he lacked the backing of state 
and federal accountability regulations, and his efforts to hold school staffs accountable for student 
results led to strong opposition from teachers. Moreover, he had to contend with a divided school 
board for much of his tenure. Hornbeck’s unsuccessful and bitter battle with the state legislature to 
boost district funding ultimately led to substantial budget deficits and his subsequent resignation 
(Boyd & Christman, 2003; Christman & Corcoran, 2002). 

Paul Vallas was hired in July 2002, 7 months after the state of Pennsylvania “took over” the 
school district. He was the first of Philadelphia’s superintendents to work within the mandated 
accountability systems established by NCLB and the state. The state takeover law specified that 
a new five-member School Reform Commission (SRC) replace the traditional school board. The 
law also required the SRC to hire a CEO, thus dropping the designation of superintendent and 
its traditional licensure requirements for the position. This served as an indicator of the enhanced 


Urban superintendents have high turnover rates. The Council on Great City Schools reports that as of 2008, su- 
perintendents in big-city school districts had been on the job an average of 3.5 years (Council of Great City Schools, 
2008/2009) An earlier study of the actual completed tenures of superintendents in 50 of the nation’s largest cities showed 
that average tenure was 4.6 years (National School Boards Association, 2002). 
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power of the position along with a recognition of the complex management challenges faced by 
the district’s chief.? 

The SRC hired Vallas because of his experience in leading improvements as CEO of the 
much-larger Chicago Public Schools from 1995 to 2001. There he had developed a reputation for 
being an energetic leader who could move quickly to carry out a far-reaching program of reform. 
His credentials as a budgetary expert—earned while a staffer in the Illinois state legislature, and 
honed as budget director for the city of Chicago—brought order to the chaotic fiscal process 
of Chicago Public Schools in 1995 and subsequently made him an attractive candidate to the 
SRC and civic leaders. This was particularly true since the Philadelphia system was still seen 
as “fiscally distressed.” Hiring Vallas—an outsider with a national reputation—was also heartily 
endorsed by governor Mark Schweiker and business and civic leaders in the city. 

In Chicago, Vallas’s accomplishments included raising student test scores (until they flattened 
toward the end of his tenure), renovating and building new schools, securing labor peace, ending 
social promotion, reaching out to neighborhood groups, providing summer and after-school 
programs for struggling students, and “cleaning up and rationalizing the bureaucratic structure” 
(Elmore, 2004, p. 111). Several of his changes provoked significant opposition, particularly the 
ending of social promotion, the use of a scripted curriculum, his attempts to reduce the power 
of the local school councils, and his focus on high-stakes testing (Coeyman, 2002; Gammage, 
2003; Pick, 1996; Russo, 2004). Toward the end of his time in Chicago, he had “worn out his 
welcome” with key stakeholders, alienating school reform advocacy organizations who objected 
to what they saw as his combative steamrolling tactics and a tendency for self-promotion (Ayers, 
2004; Gammage, 2003; Lenz, 2004; Shipps, 2006). 

In certain respects, Philadelphia posed fewer challenges for Vallas than Chicago. The school 
system was less than half the size of Chicago’s; the political scene was less complex and com- 
bative; and Vallas and his team already had experience implementing systemic change in a large 
high-poverty, racially segregated, and underfunded urban system. Vallas’s dynamic and assertive 
leadership style in Philadelphia replicated his style in Chicago, albeit with some modifications: 
He toned down his combativeness and his tendency to overreach, and he made more of an effort 
to share credit with others. He worked hard at developing collegial relationships with a range of 
stakeholders, including the SRC chair, James Nevels, teachers’ union officials, and community 
leaders. 

Within 30 days of his appointment, Vallas announced a high-speed change effort modeled on his 
work in Chicago. The Philadelphia Inquirer (“Editorial: Ambitious Agenda,” 2002) editorialized 
that he had quickly changed “the tone and substance of the conversation” on the city’s schools. 
A retrospective analysis of Vallas’s record in Philadelphia Magazine shortly after his departure 
in 2007 to become superintendent of the New Orleans Recovery District elaborated on his 
personality and leadership style: 


Vallas ...is essentially hardwired for constant momentum. He is the guy you turn to when the 
situation is ceaselessly bleak, when things need to be shaken up, torn down, reconstituted. He 
moves fast, pursuing his agenda with military urgency, chafing those accustomed to the status quo, 


3The state had already granted a short-term waiver allowing the term CEO to apply to the interim superintendent, 
Philip Goldsmith, who led the district during 2001 after David Hornbeck’s resignation. Goldsmith, like Vallas, did not 
have a background as an educator and thus did not meet state requirements for licensure as a superintendent. 
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aggressively spinning his story as he goes. He gobbles up resources, says no too little, and yes too 
much. (Fifield, 2007, p. 73) 


The city’s veteran education reporter, Dale Mezzacappa, writing on Vallas’s work in Philadel- 
phia and then in New Orleans, described him as both a “genius” and a “loose cannon,” and portrays 
the “Vallas treatment” as “do things big, do them fast, and do them all at once” (Mezzacappa, 
2008c, p. 32). His sometimes exasperating style and his proclivity to spend more money than he 
had—a surprise given his budgetary credentials—led to a deterioration of his relationship with 
the SRC and an eventual break with his key ally, SRC chair James Nevels. 

Student achievement scores rose substantially during the Vallas years (2002-2007). The per- 
centage of fifth and eighth graders (the grades consistently tested) scoring “Proficient” or “Ad- 
vanced” on the Pennsylvania System of School Assessment tests went up 26 percentage points 
in math. In reading, the percentage went up by 11 points in fifth grade and 25 points in eighth 
grade. The percentage scoring in the lowest category (Below Basic) dropped in all tested grades 
by 26 points in math and 12 points in reading. High school scores and the dropout rate showed 
only modest gains (Accountability Review Council, 2008; Neild & Balfanz, 2006).* 

Test scores continued their climb in the year following Vallas’s resignation when the district 
was led by an interim CEO who continued the same reforms. By the end of the 2007-08 school 
year, 49% of students in all grades combined scored Proficient or better in math, and about 
45% did so in reading. These scores were still low but were significantly higher than the 19.5% 
in math and about 24% in reading who scored Proficient or better in 2002 (School District 
of Philadelphia, 2008). Achievement gains occurred despite serious under-funding by the state 
(Augenblick, Palaich and Associates, Inc., 2007) and despite the city’s high and growing rate of 
poverty, the highest among the nation’s 10 largest cities (Tatian, Kingsley, & Hendey, 2007). 


SHARING POWER 


Although Vallas and his predecessor superintendents shared a passion for change and audacious 
reform agendas, Vallas worked in a substantially altered governance environment. In certain 
respects, his direct control over district improvement strategies was hemmed in by extraordinary 
constraints. The state’s governor and legislative leaders had spearheaded a state takeover of the 
School District of Philadelphia, the nation’s seventh largest system with 200,000 students, in 
December 2001. They did so with the explicit goal of tackling students’ low performance by 
privatizing the management of a significant portion of the district’s low-scoring schools and 
continuing the growth of public charter schools. 

The takeover legislation created a School Reform Commission that was given sweeping 
new powers, including the right to convert existing schools to charter schools, to outsource 
the management of schools to private providers, and to reconstitute schools with a whole new 
teaching staff. Under this law, the Philadelphia Federation of Teachers lost its right to strike 
(Boyd & Christman, 2003; Maranto, 2005; Useem, 2007). The SRC, three of whose members 


4The percentage of students in the class of 2005 who graduated in 4 years’ time improved by several percentage points 
to slightly more than 50% (Neild & Balfanz, 2006). 
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were appointed by the Governor and two by the Mayor, essentially ran the district for the six 
months after their appointment and prior to their hiring of Vallas in July 2002. 

Unlike Chicago, where the schools were an agency of the city government with Vallas reporting 
directly to Mayor Richard Daley (Shipps, 2003), in Philadelphia Vallas reported to the SRC. Mayor 
Daley had the power to facilitate Vallas’s change efforts (e.g., on matters of school construction), a 
dimension that was missing in Philadelphia. To understand the story of the reform in Philadelphia 
as it unfolded between 2002 and 2007, one has to understand the pivotal nature of the collaborative 
relationship between Vallas and the SRC. SRC chair James Nevels and Vallas were both strong and 
pragmatic leaders whose differing personalities, rather than clashing, complemented one another. 
Vallas’s brash and fast-moving style contrasted with Nevel’s deliberative and soft-spoken manner. 
Their alliance during Vallas’s first 3 to 4 years was based on their common conviction that bold 
improvement efforts had to be enacted quickly. Their eventual falling out led to Vallas’s departure 
(Boyd et al., 2008). 

Gubernatorial influence over education in Pennsylvania, always strong, intensified in the years 
leading up to the state’s takeover of the district. The takeover was orchestrated by governors Tom 
Ridge and then Mark Schweiker, both Republicans. Their allies in the Republican-led legislature 
passed laws in 1998 (Act 46), 2000 (Act 16), and 2001 (Statute 83) enabling the takeover 
once Philadelphia was declared academically and fiscally “distressed.” Democratic governor Ed 
Rendell, formerly the mayor of Philadelphia, supported the continuation of the state takeover 
after his election in 2002. Because the governor appoints the majority of SRC members and 
because the district depends on the state for more than half of its revenues, district officials must 
be extremely attentive and responsive to state political leaders whatever their political stripe. 

Although the state takeover took away the mayor’s ability to appoint all the members of the 
district’s governing body, mayor John Street (2000-2008), a Democrat, went along with the 
state takeover. He did so because it meant there would be more money for the cash-starved 
district, and because he was able to extract concessions from the governor and legislative leaders 
on the terms of the takeover. In the final agreement, Mayor Street was allowed to appoint 
two of the five SRC commissioners. He also succeeded in blocking the governor’s proposal that 
would have allowed Edison Schools, Inc., a private firm, to assume central management of the 
district. The city council also continued its role as an important player in the affairs of the district 
by virtue of its authority to approve taxes levied on the district’s behalf. These revenues cover 
35% of the district’s budget (Masch, 2007). 

The vast upgrading of federal influence over education was another way in which the gov- 
ernance environment had shifted by the time Vallas took office in Philadelphia. Congressional 
reauthorization of the Elementary and Secondary Education Act, NCLB, occurred just a few 
weeks before the state’s takeover of the district in 2001. The law gave federal officials unprece- 
dented authority over key areas of education policy (Hess & Petrilli, 2006). NCLB’s framework 
for accountability for student achievement—Adequate Yearly Progress—and the cascade of 
consequences for schools failing to meet student learning targets required district leaders and 
administrators to focus squarely on, and be held accountable for, academic performance.” New 
regulations on teachers’ qualifications also spurred efforts to upgrade the teacher workforce. 


SAbout 15% of the district’s annual budget comes from federal categorical funds from NCLB and other funding 
streams that are earmarked for services for particular groups of students, particularly low-income students. 
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To move his agenda forward, Vallas, like all superintendents, had to develop good working 
relationships with a range of interest groups and stakeholders in the city. Most crucially, Vallas 
needed to find common ground with the Philadelphia Federation of Teachers (PFT), long the 
bane of reform-minded district administrators (Boyd & Christman, 2003; Maranto, 2005; Useem, 
Christman, Gold, & Simon, 1997). The takeover had been partly caused by the desire to weaken 
the union’s power. Yet the union remained powerful despite its shrinking membership and its 
technical loss of the right to strike (Boyd & Christman, 2003). The fact that Vallas, the SRC, and 
the PFT were of one mind about reforms such as a core curriculum, smaller classes, and more 
disciplinary schools got their relationship off to a good start. 

Vallas also needed to forge collaborative relationships with the seven private organizations 
chosen by the SRC to help manage 45 low-performing schools. Governor Schweiker had wanted 
Edison to play the lead role in running both the district’s central office and as many as 60 
schools, but in a show of independence, the SRC had voted to have a “diverse provider model” 
whereby school management was outsourced to the seven groups. These included the University 
of Pennsylvania and Temple University, a regional nonprofit organization (Foundations, Inc.), a 
community-based economic development organization (Universal Companies), and three national 
for-profit education management organizations (EMOs), one of which was Edison Schools, Inc. 
Opposition by community and student groups to Edison’s presence was particularly fierce. Edison 
was assigned the largest number of schools and had been designated by the SRC to assist with 
central office strategic planning. The SRC ended up limiting EMO management scope to certain 
aspects of school operations—curriculum, professional development, school organization, and 
influence over principal selection. Consequently, EMOs did not gain control over teacher staffing 
and work rules negotiated in the PFT contract (Bulkley, 2007; Gold, Christman, & Herold, 2007; 
Maranto, 2005; Useem et al., 2006). 

One of Vallas’s first tasks was to finalize contracts with the private organizations and structure 
collaboration with them. Throughout his administration, this task was complicated by the fact that 
the organizations had ties to politically powerful officials at the state and local levels. Universities 
had significant independent clout by virtue of being major employers and “institutional anchors” 
in the city (T. Collins, 2006). The relationship with the initial seven groups—and others that were 
added to run alternative disciplinary schools—remained contentious during Vallas’s years as he 
had to make recommendations to the SRC on their funding levels and on their contract renewals 
(Mezzacappa, 2006). Vallas was not a cheerleader for private EMOs, and he tried to cut their 
funding on several occasions. But his formal power over EMO contracts was limited. The SRC, a 
majority of whom looked favorably on EMOs, controlled decisions about their role in the district. 

The constituencies supporting public charter schools had emerged as another major interest 
group by the time Vallas arrived in Philadelphia. In Philadelphia, the formal authority of the CEO 
and SRC was limited to authorizing the opening of a charter school, conducting a 5-year review 
of each charter, and renewing or revoking the charter after the review. Under Vallas, the number 
of charter schools grew from 39 to 61 schools serving more than 30,000 students (Masch, 2007; 
Zimmer, Blanc, Gill, & Christman, 2008). Charter school administrators, boards of directors, 
parents, and students were vocal champions of their schools. Many of the charter schools had 
been started by local and state politicians and community leaders of various sorts. Their political 
clout included support from SRC members and state legislators, and burgeoning parental demand 
for access to charters. The CEO and SRC felt intense pressure to open more of them, to expand 
existing ones, and to refrain from closing down those that were failing (Woodall, 2008). Even 
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though the district’s diversion of funds to charter schools represented a major system expense 
(Masch, 2007), Vallas promoted the expansion of this sector, a stance that was both politically 
necessary and increasingly in line with his own thinking. 

Finally, although community and civic organizations are not part of the district’s formal 
governance structure, Vallas had to build bridges with them. Many groups were either outright 
opposed to or skeptical of the wisdom of the state’s takeover and the diverse provider model of 
school management. He and SRC members had to legitimize these new governance structures as 
well as the broad set of reforms that Vallas initiated in the early months of his tenure. Prior to 
his arrival, student and community groups protested vociferously at SRC meetings, particularly 
when the SRC proposed to expand the number of schools being outsourced to for-profit EMOs 
to include three neighborhood high schools. The SRC’s refusal to debate these and other issues 
further inflamed the atmosphere (Boyd et al., 2008; Useem et al., 2006). Not only did Vallas have 
to quiet this turbulence, he needed to inject a sense of hope and possibility more broadly. This was 
important because Philadelphia has had a municipal culture long known for its cynical negativity 
and fragmented civic and business leadership (Whiting, 1999; Whiting & Proscio, 2007). 


WIELDING POWER: INSTRUCTION, MANAGEMENT, 
AND POLITICS 


Although the requirements of NCLB and related state legislation created a constricting framework 
within which district leaders had to operate, these laws also expanded the remedies that district 
leaders could draw on in their efforts to turn around a low-performing system. Vallas and the SRC, 
notably its chair James Nevels, seized on these new opportunities and moved swiftly to implement 
multiple strands of reform, each with its own rationale for action. Their pragmatic model of 
change was a hybrid approach that drew on ideas about market-based solutions, accountability 
mechanisms, and strong system-wide instructional guidance that were “in the air” nationally at 
the time (Boyd et al., 2008). The components of change included centralized Vallas-led efforts on 
the instructional and managerial side along with SRC-promoted privatization initiatives on charter 
schools and external school management. While seemingly incoherent from the standpoint of 
nationally debated stances on education reform, these strategies made sense on the ground in 
Philadelphia and were aimed at the common mission of improving student achievement (Henig 
& Stone, 2008). 


Instructional Reform 


In the area of instructional change, Vallas clearly had a great deal of formal authority to lead 
and he used it. The curricular expertise the district had built during the Hornbeck administration 
enabled Vallas to launch high-quality instructional and school support initiatives quickly. Unlike 
his experience in Chicago (Shipps, 2006), he moved quickly to make changes in this sphere. He 
implemented a number of top-down district-wide initiatives, the most important of which was a 
system of “managed instruction.” This approach included a common core curriculum in the four 
major subjects (that was much improved over the scripted curriculum Vallas had used in Chicago); 
formative 6-week benchmark tests that assessed students’ mastery of the material taught during 
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that block of time; new textbooks and materials; longer class periods for instruction in math and 
reading for grades K-9; and an electronic instructional management system that enabled teachers 
to access information on their students, lesson plans, and curriculum. 

Researchers have attributed gains in student achievement to these interventions (Accountability 
Review Council, 2008; Mac Iver & Mac Iver, 2005, 2006; Wang, Walters, & Thum, 2007) rather 
than to the strategy of outsourcing school management to external providers (Gill, Zimmer, 
Christman, & Blanc, 2007; Mac Iver & Mac Iver, 2006; School District of Philadelphia, 2007). 

Vallas implemented other student supports to help boost achievement: mandatory after-school 
and summer school programs and an expansion and upgrading of preschool programs (Masch, 
2007). In an effort to tackle low student performance in the middle grades, he began dismantling 
large, impersonal middle schools and moved the district toward a K-8, 9-12 grade configuration. 
Similarly, at the high school level, he launched an effort to develop smaller high schools that were 
aimed at providing a more personalized student environment (Hartmann, Maluk, & Reumann- 
Moore, 2008; Reumann-Moore & Hartmann, 2007). The number of high school choices for 
students overall went from 38 in 2002-03 to 62 during 2006-07. About half of these schools had 
fewer than 500 students. 

Other areas of instruction lagged, however, as a result of insufficient funds or uneven execution. 
The district continued to suffer from large class sizes, serious student disciplinary problems, 
insufficient attention to teacher professional development, and a weak leadership corps at the 
school level. At the high school level, reforms were slow to get off the ground. In this and other 
selected areas of action, however, Vallas failed to put together the required administrative teams 
and organizational structures, monitor them, and hold them accountable for implementing needed 
changes. 


Managerial Changes 


Vallas wielded considerable formal authority on the managerial side of his portfolio as well. 
Ever the pragmatist, he mixed and matched district-controlled efforts with those that boosted 
the role of the private sector (Gold, Cucchiara, Simon, & Riffer, 2007; Useem & Rinko, 2006). 
One strand of work involved implementing systems of accountability. Both Vallas and the SRC 
embraced the regulations and the spirit of NCLB which mainly took the form of pressuring 
and supporting school staffs to meet Adequate Yearly Progress as defined by federal law and 
state regulations. School Assistance Teams supported low-performing schools, including “walk 
throughs” by regional and central office staff and technical assistance in planning and monitoring 
improvement to student achievement. The district also implemented a system for displaying 
school and regional indicators (“SchoolStat”) at regular gatherings of regional superintendents 
and principals and at collective meetings of the regional superintendents and the Chief Academic 
Officer. 

Vallas embarked on a $1.7 billion capital campaign to build new schools, renovate old ones, and 
modernize classrooms with newer technologies and equipment. He also gave priority to human 
resource issues. He upgraded the qualifications of new teachers,° improved their retention, and 





©Between the 2001-02 and 2006-07 school years, the percentage of new teachers who were certified jumped from 
46% to more than 95%, and the number of teachers on emergency permits plummeted. Nearly one third of new teachers 
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eliminated some of the inept practices that had long characterized hiring and school placement 
of teachers (Neild, Useem & Farley, 2005; Neild, Useem, Travers, & Lesnick, 2003; Thomas & 
King, 2007; Useem, Offenberg, & Farley, 2007). The SRC, with Vallas’s support, was partially 
successful in reducing teachers’ seniority-based school transfer rights and expanding the school- 
based teacher hiring process as a result of the renegotiation of the teachers’ union contract in 
2004 (Useem et al., 2007). 

Although Vallas’s top priority was centrally directed reform of district-run schools, he also em- 
braced the expansion of public charter schools, a step that dovetailed with his own entrepreneurial 
instincts. He melded charter school development policy with long-term planning for school op- 
tions in neighborhoods, particularly at the high school level. His goal was to enhance high school 
choices for students and parents by surrounding large (and dysfunctional) neighborhood high 
schools with smaller theme-based district-run high schools and public charter high schools. Val- 
las also converted three existing middle schools to public charter schools run by a national charter 
management organization with a strong track record of school improvement. 

In another show of managerial authority, Vallas extended the “diverse provider model” that 
had been initiated by the SRC prior to his arrival. He established a network of disciplinary 
schools whose management was outsourced to three for-profit firms. Although not a strong 
supporter of contracting with external providers to run regular district schools, he did contract 
with Foundations, Inc., to manage one large neighborhood high school. Further, the district 
worked with colleges and universities and other institutional partners to design new high schools 
and improve existing ones. During his time in office in Philadelphia, Vallas (and the SRC) became 
increasingly comfortable with the notion of Philadelphia having a “system of schools”—regular 
district schools, public charter schools, and privately managed district schools—rather than being 
organized as a traditional “school system.” 

As in aspects of instructional reform, Vallas was able to use his managerial authority to chart 
new directions and build enthusiasm for his vision. When it came to execution of policies, he 
and his team followed through on changing policies and practices for teacher staffing and school 
accountability but were less effective in monitoring the work of EMOs and charter schools. The 
efforts to build new schools and renovate old ones got off the ground but were slowed by external 
constraints, including less than full cooperation from the city on site acquisition. 


POLITICAL ENGAGEMENT: MANAGING THE PLAYERS 


Vallas had to build political support for the district, particularly because neither the SRC nor its 
predecessor Board of Education could levy taxes directly. Like leaders in other big-city systems, 
Vallas and the SRC had to develop good relationships with political officials at the city, state and 
federal levels to increase budget allotments and to coordinate program planning (Cuban, 1988; 
Johnson, 1996). At the same time, Vallas had to stay on good terms with his bosses on the SRC, and 
together they had to work out a framework of collaboration and oversight of the new players with 
whom they had a formal relationship—charter schools and external school management groups. 


were hired through high-quality alternate certification routes such as Teach For America. These “intern-certified” teachers, 
who were required to pass licensure tests and to enroll in certification programs, were counted as certified by the state 
(Useem et al., 2007). 
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This process was smoothed by Vallas’s nonpartisan pragmatism, willingness to compromise, and 
extensive personal outreach to individuals in many types of roles. 


The Mayor 


A superintendent can be much more effective when he or she is backed by a politically powerful 
and engaged mayor, particularly one who has direct control over the district. This has been the 
case in recent years, for example, in Chicago, New York City, Boston, and Washington, DC 
(Reville, 2007; Wong, Shen, Anagnostopoulos, & Rutledge, 2007). Paul Vallas had a cordial if 
not close relationship with mayor John Street during the first 3 to 4 years of his term. But he 
missed the kind of direct support he had experienced from mayor Richard Daley in Chicago. In 
Vallas’s view, for example, Street was slow to expedite city-district cooperation on matters such 
as the sale of city land to the district for new school construction and the meshing of school and 
city services (e.g., child welfare, libraries, playgrounds) that he had become used to in Chicago 
(Shipps, 2006). 

Overt clashes between these two strong-willed leaders boiled over when a deficit surfaced 
during Vallas’s last year in Philadelphia. At that point, Street excoriated Vallas for poor fiscal 
oversight while Vallas accused Street of not doing enough to increase city funding of the system. 
Street ultimately worked harder to boost funds for the district and to smooth the relationship 
between the SRC and parents who were angry about their exclusion from budget decisions. The 
damaged relationship between Vallas and Street, however, was never repaired (Boyd et al., 2008). 


The Governor and Legislature 


Like other superintendents who preceded him, Vallas had to work hard to convince state officials 
to increase funds for the district each year. The state’s comparatively low tax effort and its low 
share of school districts’ costs (36%) meant that it ranked near the bottom among all states in 
its support of public education (Education Policy and Leadership Center, 2006).’ A 2007 state- 
funded study of what it would cost school districts across the Commonwealth to fund education 
adequately showed that Philadelphia’s public schools needed $3,000 more per pupil each year 
than what it was currently spending (Augenblick, Palaich and Associates, Inc., et al., 2007). 

Still, Vallas did succeed in getting more money from the state during his time in the district— 
seven percent more each year between 2002-03 and 2006-07 compared to 4% in previous years. 
His experience as a persuasive lobbyist helped him avoid the confrontational approaches that 
had led to Hornbeck’s demise in 2000. The election of Democratic governor Ed Rendell in 
2002, a strong supporter of Vallas and of public education generally, to replace Republican Mark 
Schweiker also meant that the political environment at the state level became more favorable for 
Philadelphia schools. 

In the end, despite his well-honed powers of persuasion, his pragmatic program of change, and 
support in the governor’s office, Vallas could not wrest enough additional money from the state 
and city to pay for the continuation and expansion of his reforms. Like Hornbeck before him, 


7Vallas had faced this same problem when he was in Illinois. 
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Vallas deliberately took a risk and spent more money than he actually had, an investment that paid 
off in student learning gains (Masch, 2007). By doing so, both superintendents were countering 
State legislators’ usual mantra that “more money won’t make a difference.” Budget shortfalls 
were exacerbated by partial reliance on non-recurring sources of revenue, such as private grant 
money or debt financing (a strategy that Hornbeck had used as well), and by the failure of the 
Vallas administration to exert tight controls over central-office spending. Vallas claimed that he 
was not surprised at the emergence of a deficit, but was taken aback by its size and the fact that 
it showed up sooner than he had anticipated (Snyder, 2007). The budget deficits were ultimately 
reined in after he left, but only after painful cost cutting at the school and central-office levels. 

Pressed in part by Philadelphia’s ongoing school fiscal crises, Governor Rendell and statewide 
advocacy groups mounted a successful campaign in 2007-08 to establish a more equitable school 
funding formula and boost funding levels for public schools. The long-term goal of the campaign 
is to have substantial budgetary increases for schools over a 6-year period so that they would 
approach fiscal “adequacy.” Although this came too late to help Vallas, it did mean that his 
successor, Arlene Ackerman, could begin her Ist year on the job (2008-09) without having to 
make draconian budget cuts. 


Charter Schools and External Schoo! Management Organizations 


Vallas developed working relationships with players who were relatively new to the city’s ed- 
ucational governance stage—charter school advocates, EMOs, and other partner organizations 
that had school management contracts with the district. The frostiness that had marked the rela- 
tionship between the district and public charter schools during the Hornbeck administration gave 
way to greater enthusiasm for working and planning together during the Vallas administration. 
He was especially drawn to charter management organizations (e.g., Mastery Charter, KIPP) 
that had a record of success elsewhere. But the expansion of charter schools cost the system a 
good deal of money. Indeed, charter school expenses were by far the fastest growing cost item in 
the district’s budget between 2001-02 and 2005-06 (Masch, 2007), a factor that contributed to 
Vallas’s budgetary problems. 

Vallas and the SRC did not establish strong accountability measures to guide decisions about 
renewals of school charters (Mezzacappa, 2008b). A majority of SRC members had held strong 
ideological beliefs in support of charter autonomy, and the SRC as a whole feared offending 
powerful politicians who were charter advocates (Woodall, 2008). The departure of two of the 
strongest charter supporters on the SRC in 2007 made it easier to institute stronger and much- 
needed controls during the 2007-08 school year. 

Even though he was ambivalent toward for-profit EMOs, Vallas established an office that 
removed many of the barriers that EMOs and other partner groups faced in trying to operate the 
schools whose management had been outsourced to them in 2002 (Gold, Christman, & Herold, 
2007). He avoided criticizing them publicly and did not pit one against the other, although their 
presence remained a bone of contention among many in the city. Rebuffed in his overt efforts 
to reduce the per-pupil allocation going to the for-profit EMOs, Vallas found less overt ways to 
gradually reduce by half their overall budget allocation. Yet during the Vallas years, neither the 
EMO contractors’ performance nor that of the three firms hired to run disciplinary schools was 
closely monitored (Christman, Gold, & Herold, 2005; Mezzacappa, 2006, 2008a). Contracts and 
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oversight were tightened up during 2007-08 under the interim administration and appear to be 
further tightening under CEO Arlene Ackerman. 


Parents, Civic Leaders, and Community Groups 


Vallas’s effectiveness depended not only on his ability to maneuver within unusual governance 
structures but also on his capacity to secure the support of parent and community groups. In 
this, he had a mixed record. Civic organizations, the local press, and city leaders generally 
praised his energy, optimism, and accomplishments. Even when he departed under the cloud 
that was created by his overspending on reforms, he nonetheless left with a reputation for having 
moved the district forward (“Editorial: Paul Vallas’s,’ 2007; “Editorial: Replacing,” 2007; Fifield, 
2007; Mezzacappa, 2008c). He was skilled at cultivating parents and neighborhood groups on a 
personal basis, and he spent a great deal of time with stakeholders in community settings, including 
faith-based groups (Useem et al., 2006). The district expanded its practice of contracting with 
community-based organizations to assist the reform effort in the areas of school climate and 
safety, truancy, and after-school services for students. This had the effect, however, of dampening 
the independence of these groups and their willingness to speak out about needed changes (Gold, 
Cucchiara, Simon, & Riffer, 2005). 

But even though Vallas engaged parents and community groups in a broad set of activities, his 
administration, along with the SRC, did not gather stakeholder input in a systematic or formal 
way on important issues such as the budget, discipline and school climate policies, or the renewal 
of the always-controversial school management contracts with external organizations. As the 
budget situation worsened in the spring of 2007, and as the time came for the SRC to renew the 
contracts of the school managers, parent organizations loudly protested their exclusion. Vallas 
and SRC members were late in recognizing the potential role of parent groups and other advocacy 
organizations in lobbying the state legislature and city council for increased school funding (Gold, 
Simon, Cucchiara, Mitchell, & Riffer, 2007). 


The School Reform Commission 


Perhaps most crucially, after 3 or 4 years of working well together, Vallas and the SRC parted 
ways. In the summer of 2006, two members of the SRC sparred with Vallas over budget and 
programmatic decisions and voted not to renew his contract. The contract was nonetheless 
renewed, but Vallas had to put up with more oversight from the SRC, conditions that grated on 
him. James Nevels, the powerful SRC chair, continued to support Vallas until the fall of 2006 
when the budget deficit first surfaced (Snyder, 2007, p. Al). A dispute remains as to whether 
Vallas was fired by Nevels or whether he chose to resign. 

Vallas left the district in June 2007 to become CEO of the New Orleans Recovery School 
District. At the time of his exit, he touted his accomplishments, particularly higher student 
achievement levels. At the same time, he acknowledged his defeats such as his failure to reduce 
class sizes, complete the construction and modernization of schools on schedule, and find a 
resolution to the district’s financial problems. He never lost his brashness. Responding to an 
SRC member who said that Vallas’s biggest problem was that he tried to please everyone, Vallas 
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retorted, “I promise more than I can deliver, absolutely. . . . If I promise ten things and deliver six 
or seven, I’m still ahead of the guy who promises two things and delivers one.’ He vowed not to 
stay in New Orleans more than three years because he felt that when “turn around” district chiefs 
experienced success, they ultimately became “chopped liver” because so many individuals and 
groups come to resent the credit heaped on them (Snyder, 2007, p. A1). 

James Nevels, who was openly criticized by Governor Rendell for not keeping Vallas, resigned 
2 months after Vallas’s departure. Both men, although criticized for not exercising enough 
financial oversight, were still praised in a valedictory newspaper editorial as being “the right 
people at the right time” (“Editorial: Replacing,’ 2007). The state’s budget director, Michael 
Masch, after writing a withering critique of the district’s budget processes (Masch, 2007), argued 
before public groups that the experience with the budget deficits should not obscure what the 
system accomplished. Philadelphia’s experience, he said, showed that additional resources that 
are intelligently invested yield results for students. 


URBAN SUPERINTENDENTS’ POWER RECONSIDERED 


Urban superintendents often say that their jobs are “impossible”; their ability to make change is 
undercut by school boards that are too controlling or dysfunctional, by insufficient resources, and 
by the influence of contentious interest groups, including teachers’ unions (Fuller et al., 2003). 
This examination of Paul Vallas’s tenure as CEO in Philadelphia, however, demonstrates that in the 
new standards and accountability-based environment of public education, urban superintendents 
have the power to make measurable headway in improving a district. The fact that some big-city 
districts have succeeded in raising test scores in recent years (Snipes, Horwitz, Soga, & Casserly, 
2008) reinforces the public’s expectation that a superintendent can bring about gains in student 
performance. 

In Vallas’s case, several factors were at work in explaining his enhanced authority to chart 
and execute change in Philadelphia’s schools. First, crises such as a state takeover can give 
leaders more power than they would otherwise have (Fuller et al., 2003). The mayoral takeover in 
Chicago, the state takeover in Philadelphia, and the complete restructuring of governance of the 
New Orleans schools following Hurricane Katrina have all provided “crisis” contexts in which 
Paul Vallas was given authority to undertake rapid changes as a district chief. In Philadelphia, 
Vallas and James Nevels used the opportunities for innovation that opened up in the wake of the 
state takeover, and the passage of NCLB, to act as “policy entrepreneurs,” initiating a set of bold 
changes (Blyth, 2002; Boyd et al., 2008; Sheingate, 2003). 

Second, NCLB and related state regulations on standards and accountability gave Vallas (and 
other superintendents) new powers and tools to intervene in low-performing schools to boost 
student performance. NCLB has empowered superintendents to ramp up some forms of change 
more quickly and boldly than they otherwise might have. The enactment of standards-based 
regulations at the federal and state levels has pushed political leaders to overcome partisan 
concerns and unite behind a common reform framework and has also given district leaders 
greater power to prevail over turbulent interest group politics (Useem, 2007). Of importance, in 
Philadelphia, the state takeover legislation took away the right of teachers to strike. 

Third, the results of Vallas’s efforts to raise student test scores were highly visible. In this era, 
districts post school and district scores on their Web site, often in ways that are relatively easy 
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for the public to interpret. External researchers have greater access to student data to perform 
extensive analyses, including using value-added models of students’ learning growth over time. 
This visibility, of course, can cut several ways: A district can manipulate the presentation of data 
to put the best face on discouraging scores, and district leaders can be let go when scores fail to 
budge or even go down. In Vallas’s case, test score trends. worked in his favor and helped him 
develop the support he wanted to justify and persist in his reform efforts. 

Finally, Vallas’s power was based not only on the authority inherent in his position but also 
on his personal leadership skills and his expertise in the area of district reform. Leaders who 
can inspire commitment, energize those around them, and direct people to higher standards and 
aspirations tend to be more effective in re-orienting an institution than those who lack such 
traits (Nadler & Tushman, 1990, p. 87). Big-city school systems where student achievement 
is improving appear to be led by chiefs like Vallas who have “an intense persuasive leadership 
style” (Haynes, 2007). Vallas’s energy, his ability to bring a sense of hope, and his “gangly charm” 
enabled him to build a broad-based coalition of supporters (Mezzacappa, 2008c). His ability to 
move fast also rested on his long experience in the public sector. Had the SRC appointed a 
cautious, inexperienced, or overly bureaucratic superintendent with little tolerance for managing 
complex and ambiguous situations, it is likely he or she would have been swamped by the tidal 
waves of change generated by the state takeover and NCLB. 

Arlene Ackerman assumed the CEO position in Philadelphia in June 2008. Like Vallas, her 
authority is enhanced by the state’s takeover legislation, NCLB, and by the force of her personality 
and her experience as a leader of other big-city districts. Rather than reorienting the district in 
whole new ways, she is continuing down the paths laid out by Vallas and the SRC after 2002. 
Also like Vallas, she has the support of key political figures, including governor Ed Rendell and 
mayor Michael Nutter, who assumed office in January 2008. Advocacy groups and reform-support 
organizations share a rough consensus among themselves on what needs to be done to further 
improve the schools and district operations. The darkening of the economic scene nationally and 
locally beginning in the fall of 2008, however, has cast a shadow over this relatively auspicious 
beginning. It now appears that CEO Ackerman and her team will be forced to make hard strategic 
choices about the deployment of limited resources for the foreseeable future. 
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This article offers a review of the existing historiography of the American school principal. Although 
a small and incomplete body of research, scholarship on the history of the principalship demonstrates 
that it has always been a complex and multifaceted role and that principals have historically drawn 
on shifting sources of authority to assert their institutional and personal power. The principalship has 
also been defined by who has occupied the position, as the role became increasingly dominated by 
white men in the second half of the 20th century. Recent scholarship has begun to create a more 
expansive social history of the American school principal, although a great deal of research remains 
to be done. 


In recent years, educational policymakers, funders, and researchers have become increasingly 
interested in building-level school leadership. A growing body of literature suggests that there 
is a discernible relationship between school leaders’ actions and student achievement. Although 
the effect is thought to be indirect and relatively small, studies have found that leadership can 
explain about 25% of the variability in student achievement attributed to school-level factors.! 
These research findings have dovetailed with current school reform efforts focused on site-based 
accountability to bring greater public and policy attention to issues regarding building-level 
leadership and school leaders. 

Although some researchers and reform advocates seek to extend the notion of leadership to 
encompass activities undertaken by teachers, community groups, and site-based teams, school 
leadership usually refers to the work of the principal. Thus as interest in educational leadership 
has grown, so has interest in the principalship—a position that is reportedly more difficult, 
time-consuming, and pivotal today than ever before (Center on Reinventing Public Education, 
2005; Knapp, Copeland, & Talbert, 2003; Mazzeo, 2003). Current studies and reports on the 
principalship often contrast the work of school principals today to that of school principals in the 
past and claim that the school principalship in the 21st century is, or needs to be, radically different 
from what it once was. A 2000 report produced by the Institute for Educational Leadership, a 
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student learning, effective leadership appears to play a central role. Yet because in general school-level factors only 
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variation. For a more complete review of research on school leadership, see Leithwood and Riehl (2003). 
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nonpartisan research and advocacy organization based in Washington, DC, is fairly typical in its 
portrayal of the principalship of the past: 


Being an effective building manager used to be good enough. For the past century, principals mostly 
were expected to comply with district-level edicts, address personnel issues, order supplies, balance 
program budgets, keep hallways and playgrounds safe, put out fires that threatened tranquil public rela- 
tions, and make sure that busing and meal services were operating smoothly. And principals still need 
to do all those things. But now they must do more. (Institute for Educational Leadership, 2000, p. 2) 


This characterization of the principalship of the past, however, is not based on a solid his- 
torical record. The historiography of the American school principalship is slight, but what does 
exist indicates that merely being an “effective building manager” was never “good enough” for 
principals. This is not to suggest that expectations of the principalship have not changed; they 
have. Yet the history of the school principalship makes clear that what appears most new for 
school principals today is the political environment that surrounds their work. That is, as govern- 
ment officials, policymakers, and district leaders increasingly seek to hold schools individually 
accountable for student achievement, they inevitably focus on the individual leaders of those 
schools—the principal—as agents of success or sources of failure. This is a highly individualistic 
notion of educational change, and one that has led some proponents of current school reform to 
liken school principals to “CEOs.”? As I demonstrate, however, even this notion of the school 
principal is not entirely new—although the rising market forces surrounding it may be. 

This article offers a review of the existing historiography of the American school principal. 
Although slim, as a body of scholarship it provides evidence that principals have always been 
expected to wear many more hats and fill many more roles than those described above, and that 
the principal has long been recognized as a, or even the, key player in school reform. At the 
same time, the principalship has gone through some changes. Its scale has gotten bigger; the 
professional requirements and expectations for the position have become increasingly regulated; 
and, perhaps most important, principals have drawn on shifting sources of authority to assert 
their institutional and personal power through the years. These changes have affected both who 
has occupied the position of school principal, and how principals have interacted with faculty, 
students, parents, community members, and supervisors. Yet the changes have not fundamentally 
altered the principalship. It is a role that is multifaceted, complex, and at times self-contradictory, 
but is also essentially part of the “grammar of schooling.” That is, it “has remained remarkably 
stable over the decades” (Tyack & Cuban, 1995, p. 85). Much of the existing historiography of 
the school principalship characterizes the role as subject to historical forces; this article, however, 
makes the case for placing principals in the center of analysis, as agents as well as objects of both 
constancy and change. 


THE HISTORY OF THE AMERICAN SCHOOL PRINCIPAL 


As historian Kate Rousmaniere (2007) noted, the “principalship is missing from both the political 
history of school administration and the social history of schools. It’s as if the principal did not 


?The “CEO” title for school principals is particularly prominent in charter schools. 


320 J. KAFKA 


exist at all” (p. 4).° Indeed, most historical research published on school leadership in the past 
several decades gives short shrift to the principal by examining school administration writ- 
large and focusing primarily on district-level leaders (Blount, 1998; Tyack & Hansot, 1982). 
Rousmaniere suggested three reasons as to why this has been the case: First, most histories of 
school administration have been written by nonhistorians focused on offering advice to current 
school leaders without analysis of historical evidence, second, historians have tended to lump 
school leaders and superintendents together under the somewhat limiting label of “administrative 
progressives”; and third, historians have just not been interested in the school principal because 
their personal memories are of distant individuals encountered only for disciplinary reasons. 
To this list I offer a fourth possible reason: Most educational histories tend to focus on either 
the political and institutional history of schooling or the social history of those who went to 
school and/or taught in them. With some exceptions, principals have essentially fallen through 
the middle—neither close enough to the “ground” for social historians nor far enough at the “top” 
for scholars of the politics and institutions of schooling. 

This is not to suggest that there is no existing scholarship on the history of the American school 
principal. A few educational historians began tracing the roots of the principalship in the waning 
years of the Progressive Era (Cubberly, 1934; Pierce, 1935; Reller, 1936) and in more recent years 
several scholars have begun to build a slim historiography of the U.S. school principalship (Brown, 
2005; Cuban, 1988; Goodwin, Cunningham, & Eagle, 2005; Rousmaniere, 2007). Although it 
has many holes, and is particularly thin in the postwar era, this body of literature does outline 
the basic contours of the history of the American school principal and demonstrates that the role 
has always been complex and demanding. This article considers the historiography of the school 
principal in three parts. It begins with the rise of the modern principal from the mid- 1800s through 
the early 1900s, drawing primarily on scholarship on the topic written long ago (Cubberly, 1934; 
Pierce, 1935) and supplemented by more recent histories focusing on this same era (Brown, 2005; 
Cuban, 1988; Rousmaniere, 2007). It then looks at scholarship on the more recent history of the 
school principal, which although not as detailed as the earlier works, does shed some light on 
what was expected of principals throughout much of the 20th century and seeks to place the 
expectations of the principal in broader historical context (Beck & Murphy, 1993; Brown, 2005; 
Glass, Mason, Eaton, Parker, & Carver, 2004; Goodwin et al., 2005, Hallinger, 1992). Finally, 
I conclude the article with an examination of an emerging body of literature on the American 
school principal that is beginning to create a social history of the principalship by considering 
both who the principal was and what the principal actually did in schools (Blount, 1998; Lewis, 
2006; Karpinski, 2006; Randolph, 2004; Rousmaniere, 2007; Spencer, 2002). 


The Rise of the Modern Principal 


The position of school principal as it currently exists 1s a relatively new phenomenon within 
the broader history of public education. Early schools had single teachers, or masters, who were 
answerable to the local community—often through elected or appointed school boards—for what 


31m more than 45 years prior to Rousmaniere’s publication, there had been no articles on the history of the public 
school principal published in History of Education Quarterly, the top American journal in educational history. 
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went on in their classrooms. As schools became larger in the early 1800s, and grade-level classes 
were established, the position of “principal teacher” was created. This person, almost always a 
man, was a teacher who also carried out some clerical and administrative duties that kept the 
school in order, such as assigning classes, conducting discipline, maintaining the building, taking 
attendance, and ensuring that school began and ended on time. These duties brought the principal 
teacher a degree of authority, as did his role in communicating and answering to the district 
superintendent, who tended to govern local schools from afar. As the century progressed, the 
principal teacher eventually lost his teaching responsibilities and became primarily a manager, 
administrator, supervisor, instructional leader, and increasingly a politician (Brown, 2005; Cuban, 
1988; Pierce, 1935; Rousmaniere, 2007). 

Pierce’s 1935 monograph on the history of the principalship offers a detailed look at the early 
development of the position. Based on official district reports from 12 major cities from the 
mid-1800s through the 1930s, Pierce’s work demonstrates both the mundane and complex nature 
of early principals’ work. The principal took attendance but also gained authority over the other 
teachers in his school. He worked with the broader community but also personally maintained 
school grounds. In some cities, the principal gained formal institutional power in the mid-1800s; 
in other cities it was decades before his role and authority were officially carved out. As Pierce 
noted, however, at least from a contemporary view, by the end of the 19th century, the principal 
in most large cities was recognized as a powerful and important head of the school: 


He gave orders, and enforced them. He directed, advised, and instructed teachers. He classified pupils, 
disciplined them, and enforced safeguards designed to protect their health and morals. He supervised 
and rated janitors. He requisitioned all educational, and frequently all maintenance, supplies. Parents 
sought his advice, and respected his regulations. Such supervisors, general and special, as visited 
his school usually made requests of teachers only with the consent, or through the medium, of the 
principal. (p. 39) 


The principal’s import was considered so great that in 1884 the Chicago district superintendent 
declared him “the prime factor in the success of an individual school . . . and no amount of itinerant 
supervision can supply his place” (as cited in Pierce, 1935, p. 39). Nearly 40 years later, in 1930, 
using the language of efficiency reflective of the period, the superintendent of the St. Louis 
schools made a similar declaration: 


The principal is regarded as the executive head of his school. He stands in the line of authority, and 
every element of local school control is exercised through him. Corollary to this fact the principal is 
the responsible agent in the school for all phases of management and instruction. It is the business of 
the principal to secure the best possible educational results and to do this with the utmost efficiency. 


(p. 56) 


The prestige and authority of the principal expanded in these decades for several reasons. 
First, as district bureaucracies grew, central offices in many cities were forced to hand over 
more and more responsibility and decision-making ability to school heads. As William T. Harris, 
superintendent of St. Louis schools, explained in 1871, 


When the number of pupils in a school system increases from 5,000 to 20,000, the duties of the 
superintendent and Board of Directors not only become more complex, but they change essentially 
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in quality or kind. ... When there are only 5,000 pupils the schools can be frequently visited by the 
superintendent and much stimulated by his personal presence: petty cases of discipline can be settled 
by him; he can examine methods of discipline and instruction and the proficiency of the pupils in each 
department. With 20,000 pupils this becomes impossible and the system of supervision must expand 
so as to leave the local supervision to independent principals ina large measure. (Pierce, 1935, pp. 7-8) 


The notion that principals were independent was essential. Principals were able to lead their 
schools, and to gain authority through doing so, in part because they were granted independence 
and autonomy by their superintendents. Although in later years many urban districts hired assistant 
or associate superintendents to more closely supervise local schools, bureaucratic expansion had 
already brought principals a degree of independence, and a position of leadership, that they were 
unwilling to relinquish. Thus assistant superintendents became general supervisors but principals 
remained head of their schools. Indeed, New York City local superintendents were warned in 
1904 not to infringe upon the duties and rights of principals—not only to prevent principals 
from feeling their “individuality ... interfered with” but also to ensure that they did not use such 
interference as an excuse for evading their full duty (Pierce, 1935, p. 99). 

Bureaucratic expansion was not the only reason that principals gained power over their local 
schools in the late 19th and early 20th centuries. Principals also often fought for increased 
authority. In New York City in 1859, for example, 24 grammar school principals resisted local 
superintendent examinations of pupils as unjust to teachers and students alike. They asserted 
that the exams resulted in unfair comparisons of teachers across different schools and settings 
and complained that the tests gave the superintendent “despotic” power and interfered with the 
internal management of the school (Pierce, 1935, p. 18). Elsewhere principals demanded the right 
to examine and promote students on their own, hire and fire teachers, purchase textbooks, and 
supervise the maintenance of their buildings (Pierce, 1935, pp. 40-46, 50-53). 

Third, principals gained local authority and increased their prestige by working to profes- 
sionalize the principalship. They began by creating local associations to discuss their problems 
and to respond to superintendent and board annual reports. They eventually established the 
National Association of Secondary School Principals (NASSP) in 1916 and the National As- 
sociation of Elementary School Principals (NAESP) in 1921 as distinct departments within 
the National Education Association (NEA) (Brown, 2005, p. 117). In addition to signaling 
professional legitimacy, the creation of these specialized departments within the NEA also 
indicated that principals drew upon specific knowledge and skills at a time when scientific 
inquiry was considered an essential aspect of any profession. As Pierce documented, the estab- 
lishment of professional associations led to sponsored research on the principalship, whereas 
local associations instituted professional yearbooks, monthly journals, and group and individ- 
ual investigations into problems of practice (Pierce, 1935, pp. 193-196, 204-205). Professional 
associations also encouraged states to pass laws requiring greater certification requirements for 
principals, thereby elevating the status of the principal by establishing academic qualifications 
and specialized expertise as prerequisites for the position (Brown, 2005, pp. 118-119). Thus, 
although principals were once seen as teachers who received better pay and additional respon- 
sibilities, the principalship gradually came to be seen as a distinct and increasingly prestigious 
profession. 

The fourth way that the principals raised both their status and their authority in the 1800s and 
early 1900s was by increasing their supervisory position over teachers. As their teaching roles 
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waned—by the mid-1930s 70% of urban elementary school principals had no teaching duties 
(Rousmaniere, 2007, p. 11)—principals increasingly came to be viewed as teachers of teachers. 
A professor investigating New York City schools explained this idea in a 1910 report: 


In brief, the principal, as a professional leader, is working for the same ends as the classroom teacher, 
but his pupils are the teachers themselves. His worth is to be judged, primarily, by his skill as a leader, 
as a teacher of teachers, just as theirs is to be judged by their skill as a teacher of children. Thus, his 
first duty is to his teachers, to help them grow professionally (as cited in Cuban, 1988, p. 61). 


The notion that principals were in charge of other adults in the building, and responsible 
for their professional growth, brought the position added prestige and power. It may surprise 
some readers to learn that the model of classroom visitations promoted by superintendents and 
school boards in the late 1800s was markedly similar to what is used today: Principals were 
expected to observe classroom lessons daily, participating at times to have their influence felt. 
They were also expected to conference with teachers, provide personalized instruction and advice 
on ways to improve teaching practices, and ultimately rate teachers on their performance and 
the performance of their students (Pierce, 1935, pp. 71-75). As John Philbrick, superintendent 
of the Boston schools, explained in 1877, “Merely looking on and seeing teachers teach is not 
the supervision of instruction which is to be expected of a principal” (as cited in Pierce, 1935, 
p. 71). Moreover, as early as 1863 superintendents worried that principals’ other duties were 
distracting them from their ability to spend time working with struggling and new teachers. 
In 1873 the superintendent of New York City schools recommended relieving principals of 
many of their clerical duties so that they could spend more time in classrooms (Pierce, 1935, 
p. 30). 

Principals’ supervisory responsibilities thus helped raise the prestige of the principalship— 
not only by putting principals in charge of teachers (and allowing them to rate, hire, and remove 
teachers) but also by sharpening the distinction between principal and teacher. As Andrew Abbott 
(1988) argued in The System of the Professions, professional jurisdiction has historically been a 
competitive game. In this case, principals won out over other adults in the building, increasing 
the status of the principalship at the expense of teachers. 

Finally, principals were able to raise their status and authority by establishing themselves as 
local leaders. Although their initial community obligations were confined to notifying parents 
of student attendance and performance, and receiving formal notifications from area truancy 
officers, by the late 1800s principals had organized mother clubs and social functions to gain 
support and trust from parents (Pierce, 1935, pp. 123-127). Principals in New York began holding 
precursors to today’s ‘“‘back-to-school nights” or school open houses in 1904, running seminars on 
subject areas ranging from geography to manual training (Pierce, 1935, p. 132). Home and school 
associations became commonplace in many cities in the early 1800s (Cutler, 2000, pp. 34-40), 
often with principal initiation (Pierce, 1935, p. 133). By the mid- 1900s, principals were extending 
their reach beyond parents and becoming involved in local social and health services—such as 
societies for indigent children. Their reach grew particularly wide during World War I, as schools 
became the center of a variety of charity drives and civic activities (Pierce, 1935, pp. 142-143). 
Indeed, in many cases principals were able to use their position as head of school to become head 
of community. As Cuban (1988) explained, their efforts in this regard were both practical and 
strategic. 
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Since the birth of the position, principals have spent a portion of their time organizing parent groups, 
speaking at weekly business club luncheons, meeting with influential citizens to help subsidize a 
school project and similar activities. Called “community relations,” or, later, public relations, these 
acts were implicitly political in their search in the community for public support of the principal’s 
authority. Many principals simply recognized a central fact in American life: any public organization 
_..in a democratic society will not be left to the experts alone. (p. 76) 


By the 1920s, the modern school principalship had been established and looked markedly 
similar to the position today: Principals had bureaucratic, managerial, instructional, and commu- 
nity responsibilities. They were expected to lead and instruct teachers, to monitor students, to 
communicate with the district, and to work with parents and members of the wider community. 
Moreover, they were seen as pivotal figures in any school reform effort. For many observers at 
the time, the principal was the school. 


The Principalship of the 20th Century 


The rise of the modern principal did not happen in a historic vacuum, as more recent scholars of 
the history of the principalship have intently pointed out. At the same time that principals were 
working to raise their prestige and authority, schools were increasingly replacing the church as 
American society’s central site of socialization. School enrollment more than doubled between 
1870 and 1898 (from 7,000,000 to 15,000,000), and by the turn of the century 71% of Americans 
between the ages of 5 and 18 were enrolled in some schooling—averaging 5 years of attendance. 
Most parents sent their children to school out of choice rather than compulsion; attendance laws 
were not regularly enforced for another several decades. By 1940, when compulsory education 
laws had extended the number of years American youth were expected to attend school and, 
of importance, when local officials were more regularly enforcing these laws, nearly 80% of 
youth between the ages of 14 and 17 attended some high school, with more than 50% becoming 
high school graduates (Tyack, 1974, pp. 66, 183). This meant that as more and more youth 
attended school for more and more years, education became an increasingly important part of 
American life, and teachers and school principals became even more important figures in their 
local communities. 

Of course local communities were changing during this time as well. Industrialization, urban- 
ization, migration and immigration brought families pouring into the nation’s cities, decimating 
many small towns and remaking the face of America. As Tyack and Hansot (1982) explained, 
many political and educational leaders viewed the influx of non-Anglo-Saxon immigrants into 
American cities through the prism of their own small-town, Protestant upbringing and did not 
hesitate to call on the government and schools to instill in these new citizens the values and 
cultural norms that they had grown up with themselves. Stanford University professor Ellwood 
P. Cubberley, for example, wrote in 1909 that the new immigrants were “illiterate, docile, often 
lacking in initiative, and almost wholly without the Anglo-Saxon conceptions of righteousness, 
liberty, law, order, public decency and government,” and thus needed to learn to adapt to Amer- 
ican ways (as cited in Tyack & Hansot, 1982, p. 117). Schools were the central site for such 
Americanization, and this social project lifted education’s—and principals’-—importance even 
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further. School leaders gained prominence in American communities in tandem with the rise of 
education itself. 

In the past few decades, several nonhistorians have taken up the task of placing the 20th- 
century history of the school principal in broad context—primarily by focusing on how large 
historic shifts have been reflected in expectations of the school principal. Lynn Beck and Joseph 
Murphy’s (1993) Understanding the Principalship: Metaphorical Themes, 1920s—1990s has 
been highly influential in this regard. In examining “the language used in talking about being a 
principal” (p. 3), Beck and Murphy used an organizational frame for understanding the history 
of schooling and then applied it to the role of the principal. Their portrayal of the principalship 
has been followed by several scholars (Brown, 2005; Goodwin et al., 2005), who use Beck and 
Murphy’s periodization and metaphors to frame their own work. Others have approached the 
history of the principalship from a similar lens: creating analytic categories to explain what was 
expected of principals based on sources like administrative textbooks (Glass et al., 2004) or policy 
and research reports (Hallinger, 1992). Taken together, this work provides an expansive analysis 
of the social construction of the principalship through time but offers little historical data on what 
principals actually did, or on how historic events or movements impacted local communities and 
schools themselves. Nonetheless, this scholarship does reveal the complex nature of the school 
principalship in the 20th century, at least as far as the position was understood by intellectuals, 
academics, policymakers, and social commentators. 

Implicit in all of these works is the notion that changing expectations of the school principal 
reflected changing expectations of schooling more broadly. Principals’ sources of power shifted 
along with expectations of what schools and school leaders were expected to do. Beck and Murphy 
(1993), for example, asserted that in the 1920s and 1930s principals were considered spiritual 
and scientific leaders, as both the church and scientific management played important roles in 
American political life. The 1940s, however, brought World War II and fears of fascism and 
communism, and principals’ role as a democratic leader was elevated over previous expectations. 
The principal remained a manager and supervisor, but according to Beck and Murphy there 
was a greater expectation that other members of the school community—including faculty and 
even students—would help make decisions and govern the school. In this sense, the principal’s 
authority was drawn somewhat from his role as a democratic leader. This form of participatory 
management, however, was limited, and Beck and Murphy asserted that in the postwar decades, 
emphasis returned to the principal as an administrator. In fact in the 1950s efficient administration 
was emphasized so much that principals were instructed on minute and even trivial duties such as 
how custodians should be introduced to students and how faculty lunchroom conversations should 
be structured (p. 59). Yet Beck and Murphy also made clear that principals were still expected to 
function as instructional leaders and use the professional training they received and the scientific 
theories they were presumed to have mastered to bring about the very best classroom teaching 
and learning (pp. 73-76). 

Hallinger (1992) argued that in the 1960s and ’70s principals were also expected to manage 
federally sponsored entitlement programs and curricular initiatives, and maintained that “as a 
result of increased federal intervention in local policy, principals came to be seen as potential 
change agents” (p. 2). Hallinger noted that this focus on the principal’s capacity to enact change 
was only affirmed in the 1980s with the popularity of Ron Edmonds’s effective schools research, 
which emphasized that strong administrative leadership was acommon characteristic of successful 
schools. Thus principals were once again called upon as individuals to be instructional leaders 
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and to be the “primary source of knowledge for the development of the school’s educational 
programme” (p. 2). 

Works by Hallinger, Beck and Murphy, and others are helpful in demonstrating that principals 
have always been expected to be ‘nstructional leaders, even as the language and buzzwords 
surrounding instructional tasks have changed, and that their roles have always represented a 
mixture of expectations and competing demands. Yet while painting broad strokes on the history 
of the American school principal, this literature leaves out quite a bit. Specifically, in ignoring 
how race, class, and for the most part gender, have impacted schooling and society throughout 
the 20th century, these authors ignore huge shifts in the history of the principalship.* In the 1950s 
and ’60s, for example, at the same time that efficiency, scientific knowledge, and the shoring 
up of democratic institutions to fight communism were central to American schooling, the U.S. 
Supreme Court’s Brown decision and the larger civil rights movement were affecting American 
public education as well. In many cities and towns in both the North and South, questions of what 
schooling was supposed to accomplish and who it was intended to serve raised doubts about local 
principals’ authority. In the South, desegregated school systems often eliminated Black principal 
positions, so that Black men and women would not be supervising Whites (Fultz, 2004; Karpinski, 
2006; Tillman, 2004). In the North, community and student protests often openly challenged White 
principals’ authority and at times called for Black school leaders to replace them (Dougherty, 
2004; Kafka, 2008; Mirel, 1993; Perlstein, 2004). Indeed, the notion of “principal as community 
leader,’ which Beck and Murphy (1993) asserted was greatly emphasized in the 1970s, and the 
belief that “community power [was] a coming reality,” as a textbook they cited explained (p. 116), 
were ideas born out of struggles and conflicts that are missing from this literature on the history 
of expectations of the principal in 20th-century America. Recently, however, a new generation 
of historians has begun to piece together a social history of the school principal that incorporates 
the lived history of principals, schools, and communities. 


The Making of a New Historiography 


Rousmaniere (2007), in an article directed at historians of education, called attention to the dearth 
of historically grounded scholarship on the principalship and began to point to a social history of 
the principal by drawing on both primary and secondary source data about what principals did 
and who they were throughout American history. While focusing on the late 18th and early 19th 
centuries, Rousmaniere made clear that who principals were affected the position as much as 
any formal declaration of their job requirements and level of authority. That is, the principalship 
became an increasingly prestigious position distinct from that of teaching as the role became 
increasingly defined as White and male. 

Indeed several historians have documented the marked downturn in the number of women and 
Black school principals in the second half of the 20th century and have sought to understand why 
these trends occurred (Blount, 1998; Fultz, 2004; Karpinski, 2006; Shakeshaft, 1989; Tillman, 
2004). The decimation of the Black principal force in the South following the 1954 U.S. Supreme 
Court Brown decision was particularly pronounced. Karpinski provided statistics on the quick 


4 Within this body of literature, Brown (2005) is an exception. The chapter devotes considerable attention to the history 
of the female school principal as distinct from the larger historiography. 
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and furious elimination of Black principalships in the South once school desegregation got 
underway. In Kentucky, for example, the number of Black principals dropped from 350 to 36 
between 1954 and 1970, and nearly all who remained were employed in Louisville. In Maryland, 
a border state, there was a 50% reduction in Black principals at the secondary level and a 27% 
reduction overall—at the same time that White principals increased 167%. In North Carolina, 
Black principals declined by 95% between 1963 and 1970. As Karpinski pointed out, these job 
losses were not only traumatic for those involved, but “contributed to drying up the ‘pipeline’ 
of African American teaching recruits by eliminating administrators” who had cultivated and 
recruited Black educators and who had “traditionally buffeted staff from outside distractions and 
promoted advancement of future leaders” (p. 251). Particularly for segregated Black communities 
in the rural South, the elimination of the Black school principal also constituted the elimination of 
a local leader who served as head of a school and often as head of the community. Moreover, the 
effects of the demise of the Black principal are still with us today. As of the 2003-04 school year, 
only 9.3% of all school principals in the country were Black (National Center for Educational 
Statistics [NCES], 2007). Data from 1998 indicate that even in urban schools with predominantly 
minority populations, nearly two thirds of principals were White (Tillman, 2004, p. 137). 

If the Black school principal (at least in the South) was eliminated primarily in the aftermath 
of Brown, the story of the decline of the female principal is more complex. Although, as indicated 
earlier, the position of school principal began as an almost completely male position, by 1905 
nearly 62% of elementary school principals were women. This was in comparison to less than 6% 
of secondary school principals (Tyack & Hansot, 1982, p. 183). The reasons for the distinction 
were clear: Women were more likely to be found in positions supervising women than supervising 
men, and were more likely to occupy lower status and lower paying supervisory positions. As 
Rousmaniere (2007) explained, the elementary school principalship fit both these requirements. 
Not only were elementary school teachers almost exclusively women at the turn of the last century, 
but elementary school principals were more likely to have teaching and lunchroom duties than 
their secondary counterparts, and less likely to have clerical help, assistant principals, and their 
own conference rooms. Unlike elementary school principals, high school principals were usually 
required to hold advanced degrees and certifications that were often difficult for women to obtain, 
and they were paid considerably more for their work (pp. 16-17). 

Nonetheless, elementary school principalships provided women leadership opportunities, ele- 
vated positions in the local community, and the opportunity to be appointed to other supervisory 
positions—including, occasionally, to district and associate superintendencies and directorships. 
By 1973, however, fewer than 20% of elementary school principals were women, and less than 
2% of high school principals were female (Tyack & Hansot, 1982, p. 183). The reasons for this de- 
cline are less clear-cut than with Black principals, whose jobs were eliminated quickly and openly 
in the context of desegregation. Blount (1998), in her study of gender and the superintendency, 
traces the roots of women’s declining representation in educational school leadership positions 
to several causes. School consolidation clearly played a role. As rural elementary schools were 
combined, making them bigger and more bureaucratically organized, they became, as one ob- 
server at the time noted, “large enough to hold positions . . . attractive to males,” and women were 
increasingly structured out of jobs (as cited in Blount, 1998, p. 123). Yet there were other reasons 
for the decline in the female principalship. A perceived crisis of masculinity both during and 
following World War II led local schools, states, and the federal government to recruit more men 
into schools. To make what was considered a lowly female job seem more attractive, men were 


328 J. KAFKA 


often promised quick ascendancy to administrative positions, where they could earn more money 
and enjoy higher status (pp. 115-116). The G.I. Bill also played an indirect role in the decline 
of female principals, as it opened the door for millions of men to enroll in college and graduate 
school. Some of these men earned advanced degrees in educational administration—degrees that 
were becoming increasingly difficult for women to obtain. Today women comprise nearly half of 
all school principals in the United States but remain underrepresented as high school principals. In 
the 2003-04 school year, only 26% of public secondary school principals were women, compared 
to 56% at the elementary level (NCES, 2007). 

Taken together, these works on race and gender and the school principalship begin to paint a 
more expansive and more nuanced picture of who the school principal was (or was not) throughout 
American history. In addition, a growing body of biographical studies and oral histories of 
individual principals has begun to bring together more recent history of the principalship through 
detailed storytelling of individuals’ day-to-day work and experiences (Brubaker, 1995; Lewis, 
2006; Randolph, 2004; Reynolds, 2002; Spencer, 2002). Much of this scholarship takes race, 
ethnicity, class, and gender into account, as these issues played important roles in individuals’ 
lived experiences. A great deal of room remains, however, for current and future scholars to 
continue and expand upon this work. Much about what principals did and how they interacted 
with others throughout American history is still unknown. Were principals able to straddle the 
demands placed on them as bureaucrats, instructional mentors, building managers, and community 
leaders, for example, or did they tend to emphasize one role to the detriment of others, as current 
policy reports imply? Did principals’ response to these demands vary by race, gender, school 
location or school characteristics, as we might assume, or did some commonalities prevail? Only 
real historical investigation will be able to answer these questions. 


THE PRINCIPALSHIP TODAY 


Goodwin et al., in their 2005 review of the history of the secondary school principal, argued 
that as principals have been asked to do more and more tasks and take on more and more 
responsibilities, their role has become an “accumulation of expectations that have increased the 
complexity of the position until it has reached a bifurcation point where change is inevitable” (pp. 
2-3). Yet the change they forecast sounds markedly similar to what has come before: Principals 
should be strategic, instructional, organizational, political, and community leaders, and they 
should be afforded the autonomy commiserate with their responsibilities. There is no question 
that principals today are asked to do a great deal—as are schools. The No Child Left Behind 
Act, and similar measures from states and cities, demands that educators be held accountable for 
student achievement at a school and classroom level. Principals experience that accountability 
pressure in deeply personal ways—particularly in high turnover cities like New York where many 
administrators are new to the position and bereft of the connections and sources of authority that 
principals may have been able to call on in the past (see Shipps & White, 2009/this issue). Yet the 
history of the school principal demonstrates that although specific pressures might be new, the 
call for principals to accomplish great things with little support, and to be all things to all people, 
is certainly not. What is new is the degree to which schools are expected to resolve society’s 
social and educational inequities in a market-based environment. 
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As principals are asked to compete for students, parents, and community support, and risk 
losing students (and thus funding) if their test scores do not improve, principals’ individual 
importance in the success or failure of a school has seemingly increased. Historically, however, 
the role of principal itself has not radically changed, even as the political environment surrounding 
it has. The historiography of the school principalship suggests two reasons why this is the case. 
First, principals’ position within district bureaucracies has remained relatively stable. They are 
neither at the top of the educational hierarchy nor at the ground level, but are somewhere in the 
middle, answering to district officials, policymakers, parents, students, teachers, and community 
members—just as they have been, to varying degrees, since the 19th century. Thus although 
principals’ status and pay rate continue to be higher than those of the teachers they supervise, 
they remain constrained by federal, state, and local dictates beyond their influence. Second, 
the fundamentals of schooling have not changed, and thus neither have the demands placed on 
principals’ time and attention. They remain managers, administrators, supervisors, instructional 
leaders, and politicians. 

The current market environment may place more emphasis on the political aspects of the 
principalship than in the past, but the notion that principals are accountable to, and somewhat 
dependent upon, public opinion is not new. Indeed, the history of the American school principal 
suggests that today’s focus on individual leaders as enactors of building-level change may yield 
mixed results. On one hand, principals have often been central to efforts to improve schooling 
and enact educational change. On the other hand, principals have historically acted as both part 
of and in response to existing structures and systems, and as long as those remain fundamentally 
the same, so, too, will the work of the American school principal. 
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This article critically reviews two bodies of literature that potentially share common concerns, yet 
rarely overlap: distributed leadership and educational micropolitics. Alternative explanations for 
the split between these two analytical approaches to school organization are explored in sections 
on problem framing, methodology, and the marketplace of educational research and publishing. The 
article concludes that the separation between explorations of educational micropolitics and distributed 
leadership is both consequence and reflection of an increasingly prevalent managerialism required 
in the current policy context, which emphasizes smoothing out micropolitical conflict rather than 
examining it or learning from it. 


The study of educational politics foregrounds disagreement. Researchers identify the ways 
that various stakeholders offer competing claims on scarce resources and attempt to influence 
policy formation and implementation at different times and at various levels of governance. 
Examinations of educational politics focus attention on our contradictory aspirations for schools 
as well as disagreements about the means to achieve the goals that are set for them. Understanding 
educational politics requires examining how, why, and with what result conflicts emerge. As 
Iannaccone (1991) put it, “A politics perspective’s unique focus is on processes for producing 
policy from conflict” (p. 467). 

The articles in this collection demonstrate that the scholarship on educational politics en- 
compasses a wide range of levels of analysis: Some work examines federal policymaking and 
implementation; other articles look at the intersection of municipal governance and local school 
district administration. The focus of this article is within-school politics, a large area of study 
which overlaps with the literature on school change and reform or, as Mawhinney (1999) put 
it, “the problems arising when macro directions meet micro realities” (p. 159). The study of 
politics within the school—micropolitics—is sometimes understood as the study of how things 
really work, not how an organizational chart or a principal’s action plan would like them to work. 
Marshall and Scribner (1991) identified power relationships, conflict, and the policy process as 
concepts central to the study of political dynamics that exist within schools (p. 349). According to 
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Hoyle (1982) micropolitics “is characterized more by coalitions than by departments, by strate- 
gies rather than by enacted rules, by influence rather than by power, and by knowledge rather 
than by status” (p. 88). 

The study of educational leadership at the school site would seem to provide a rich opportunity 
to examine conflict and to explore the ways that ideals and realities or plans and implementation 
diverge: Teachers and administrators do not always share a vision of schooling or work collabo- 
ratively; educators and parents are often mutually suspicious and sometimes antagonistic; school 
site reform plans shift and change over time because of the specific people involved. The study of 
school level educational leadership through the lens of micropolitics has the potential to generate 
interesting and potentially useful analyses of the different experiences and expectations of those 
closest to educational policy implementation. 

Nevertheless, research into distributed leadership, a significant and mostly new body of ed- 
ucational research, has been much more likely to avoid micropolitical analyses than to conduct 
them. The distance between micropolitics and educational leadership can be illustrated in this 
shorthand way: Two collections of work that both reflect and examine the growing significance 
among educational leadership researchers of “distributed leadership” have recently been pub- 
lished (Leithwood, Mascall, & Strauss, 2009; Spillane & Diamond, 2007).! Both books provide 
a good foundation for understanding the empirical and normative claims being made in the name 
of distributed leadership; both include detailed case studies as well as broad diagnoses of the 
legitimacy of the new theoretical terrain being mapped. Although both books are concerned with 
the way that leadership is stretched across roles and positions, and both books include empirical 
investigation of the ways that leadership is enacted by different individuals at school sites, neither 
includes an index reference for “conflict” or for “politics”; micropolitics garners mention on 2 
pages out of almost 500. Why does a literature that focuses on individuals’ work within school 
sites and that investigates the different ways schools are managed and their implications fail to 
feature a micropolitical analysis? 

In this article I attempt both to substantiate the claim that micropolitics and distributed lead- 
ership occupy surprisingly separate spaces in the literature on school leadership and to provide 
some ways of understanding it. I examine the distributed leadership literature and illustrate how 
it has addressed and avoided micropolitics. I also examine the micropolitics literature for poten- 
tial points of connection. I then attempt to delineate the methodological, epistemological, and 
conceptual explanations for the divide between these two bodies of literature. 


AN OVERVIEW OF RECENT RESEARCH ON DISTRIBUTED 
LEADERSHIP 


It is not fair to review a body of literature solely to criticize it for what it isn’t. It is fair, however, 
to ask why this body of work has grown and developed in the way that it has and to attempt 
to puzzle out why work with such popularity, empirical foundations, and conceptual versatility 
seems consistently to shy away from political questions. Politics is concerned with interests: In 


'T am coauthor of a chapter in the Leithwood, Mascall, and Strauss collection. Consistent with a commitment to 
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schools, where leadership is always distributed in some way, what happens when interests of 
different leaders, or of leaders and followers, don’t converge? How are micropolitical alliances 
and coalitions mobilized to build consensus? Does the distributed leadership literature smooth 
out disagreements and assume uniformity? Does the literature perform a uniformity of interests, 
by neglecting the very topic? In this section, I review some of the recent literature on distributed 
leadership and examine the ways that micropolitical analyses are and are not utilized. 

Distributed leadership is an increasingly popular concept associated with a growing body 
of empirical and normative writing in the field of education. Like many educational labels be- 
fore it—“restructuring,” for example—“its conceptual elasticity is considerable” (Hartley, 2007, 
p. 202). One review of the literature summarized, “We found little agreement as to the meaning 
of the term ‘distributed leadership” (Bennett, Wise, Woods, & Harvey, 2003, p. 2). As two of 
the authors most closely associated with this work state, 


The appeal of a distributed perspective lies partially in the ease with which it becomes many things 
to many people. Frequently used as a synonym for democratic leadership, shared leadership, collab- 
orative leadership, and so on, the distributed perspective easily and effortlessly entered the discourse 
about school leadership and management. (Spillane & Diamond, 2007, p. 1) 


The flexibility of the concept explains some of its growing prevalence in discussions of school 
organization and administration. Another explanation derives from the fact that, in times of 
increasing accountability and the corresponding attention to leadership generally, 


distributed leadership, . . . it is argued, also enhances opportunities for the organization to benefit from 
the capacities of more of its members, permits members to capitalize on the range of their individual 
strengths, and develops among organizational members a fuller appreciation of interdependence and 
how one’s behavior affects the organization as a whole. (Leithwood et al., 2009, p. 2) 


Ina policy environment that features “leadership” as a self-evident good, distributed leadership 
is even better. As the editors of a recent collection observe, “the overwhelming disposition of the 
contemporary, normatively oriented literature on distributed leadership is enthusiastic optimism 
about its anticipated benefits” (Leithwood et al., 2009, p. 2). 

Distributed leadership is a popular concept with many overlapping definitions. So just what are 
some of these definitions around which most work seems to cohere? “Distributed leadership . . . is 
group activity that works through and within relationships, rather than individual action” (Bennett 
et al., 2003, p. 3). A distributed perspective “is not a model but a conceptual lens or diagnostic tool 
for probing and analyzing practice” (Spillane, & Diamond, 2007, p. 148). Building on the ideas 
of distributed cognition, distributed leadership describes the ways that leadership activities are 
stretched across different people and positions within organizations, and where the result is greater 
than the sum of the individual parts (Gronn, 2000, 2002; Spillane, 2006). It can be understood as 
an organizational, not individual, property (Ogawa & Bossert, 1995) where leadership work “is 
done redundantly and sometimes jointly [and] it is hard to find any one person who is responsible 
for the accomplishment of change” (Heller & Firestone, 1995, p. 84). Although the research 
literature remains agnostic about impact—“there are no empirical data on the effectiveness of 
distributed leadership, in terms of pupil or student achievement” (Bennett et al., 2003, p. 2)—and 
although there is striking scholarly agreement that studies of impact are needed to move the field 
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beyond its current “adolescence” (Leithwood et al., 2009 p. 269) or “preadolescence” (Spillane 
& Diamond, 2007, p. 163), some writing aimed at practitioners or reformers nevertheless asserts 
causality. For example, one work, The Distributed Leadership Toolbox, states in its preface, 
“Distributed leadership is a method for engaging educational practitioners in meaningful and 
timely dialogue about the effectiveness of their leadership practices as they relate to enhancing 
and changing classroom practices. This, in turn, enhances student learning” (McBeth, 2008, 
p. Vii). 

This conceptual tension runs throughout the literature, sometimes within the same work. 
Leadership is always distributed in some way within organizations, and consequently using a 
distributed framework is a descriptive task. “Distributed” leadership is also said to be something 
that schools should want more of and policymakers should promote. For example, The Distributed 
Leadership Toolbox includes both a CD and more than 50 pages of reproducible templates 
for school leaders’ use yet begins its second chapter with a quote from Spillane: “Distributed 
leadership is not a blueprint for practicing school leadership more effectively” (Spillane, as cited 
in McBeth, 2008, p. 11). 

Given the variety of ideas and purposes that fall under the label “distributed leadership,” 
Mayrowetz (2008) concluded that “one universal usage of distributed leadership may never be 
achieved, and perhaps it is unadvisable to seek it given the proliferation of definitions that have 
emerged” (p. 433). He usefully identified four common uses of the term “distributed leadership,” 
“which include the original descriptive theoretical lens and three prescriptions for how sharing 
leadership in schools can improve practice” (p. 424). They range from a “theoretical lens” (p. 425) 
to an antidote to the problems of school administration hierarchies (p. 438) and include greater 
“efficiency and effectiveness” (p. 429) and “distributed leadership as human capacity building” 
(p. 431). Given this diversity of approaches, I clarify my own use of terms here: In this article, 
the work I am drawing from utilizes the label “distributed leadership” differently, but I make 
claims about “the distributed leadership literature” in the singular. I do so not to suggest that this 
large and internally inconsistent body of work is uniform but because this is how the authors 
themselves have classified their work. 

Research that explores “distributed leadership” often uses case study methodology where 
data collection strategies combine interviews and observations. Consistent with the norms of 
the field of educational administration, some distributed leadership studies rely on administra- 
tor self-reports or utilize larger scale survey research (e.g., Camburn, Rowan, & Taylor, 2003). 
Other work combines both; one noteworthy mixed methods study utilized sampling logs for 
52 school principals, collecting data on principals’ work at random times throughout the day 
(Spillane, Camburn, & Pareja, 2007). Causal or inferential claims about outcomes for stu- 
dents can not yet be substantiated rigorously and “the lack of serious effort ...to assess the 
contribution of greater leadership distribution to the long list of desirable outcomes typically 
invoked by advocates” is cause for “disappointment” (Leithwood et al., 2009, p. 280). Some 
researchers have called for methodological changes—valid and reliable instruments and larger 
scale investigations—to build upon the current descriptive foundation (Spillane & Diamond, 2007, 
p. 165). 

Distributed leadership writing often includes built-in criticisms of the state of the current 
research. For example, in a study of Texas elementary schools, researchers “seeking implications 
for democratic school governance” (Maxcy & Nguyen, 2006, p. 163), criticize existing distributed 
leadership frameworks for the elision of distributed leadership with “task delegation,’ neither 
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of which is the same as “leadership enacted through democratic processes—processes engaging 
stakeholders in collective and deliberative determination of means, modes, and ends in schooling” 
(p. 164). This kind of critique asks researchers to pay greater attention to how, by whom, and 
to what ends leadership is distributed; such decisions are not ideologically neutral. Hatcher 
(2005) examined the relationship between the rhetoric of distributed leadership and the reality of 
managerial power. In her study of distributed leadership in a school improvement context in New 
Zealand, Timperley (2005) observed that there was a normative push in some of the literature 
for distribution of leadership activities without consideration of the quality of those activities. 
She concluded that this normative stance was more than naive; it was also potentially negative. 
She wrote, “Distributing leadership over more people is a risky business and may result in the 
greater distribution of incompetence” (p. 417). Researchers investigating distributed leadership 
utilize multiple definitions of their terms and also debate the central concepts. Yet very little is 
said about school level micropolitics. 


The Absence of Micropolitical Analysis 


In work that asserts its central question is, “What does it mean to take a distributed perspective?” 
(p. 2), Spillane and Diamond (2007) answered with virtually no explicit attention to politics, 
micropolitics, or conflict. The way their inquiry has been framed around “three broad categories: 
compass setting; human development; organizational development” (p. 3) circumvents what other 
scholars locate as central to studies of leadership, management, and professional community, 
namely, the way that goals, purposes, and strategies are contested (see, e.g., Blase, 1995; Blase 
& Blase, 1997; Malen, 1995; Mawhinney, 1999). 

Is the absence of specific mention of micropolitics simply a matter of using different vocab- 
ularies for studying the same thing, or have, in fact, scholars of distributed leadership neglected 
something so central as micropolitics? Spillane and Diamond (2007) stated that “in efforts to 
understand the practice of leading and managing schools, we must pay close attention to the 
interactions, not simply the actions” of individuals (p. 6). In other words, they acknowledge that 
the social processes at work involve give and take and that, by definition, leadership will involve 
not just the decisions of leaders but also those of followers, “because social influence is a two-way 
affair” (p. 9). However, when obvious links to the micropolitical work of leadership could be 
drawn, they are not. For example, they rely on Larry Cuban’s (1988) leadership work to anchor 
their discussion of the overlap between management and leadership (Spillane & Diamond, 2007, 
p. 153), but never mention the fact that the very citation they’ve used included three imperatives at 
the heart of leadership: instructional, managerial, and political. In another chapter in the Spillane 
and Diamond volume, Halverson (2007) demonstrated that he is aware that a focus on leadership 
distribution might overemphasize routines and structures and underemphasize the people who 
work with and within them. He concluded his chapter with a response to a rhetorical question: 


Does reliance on the analysis of routines as the path to professional community give short shrift 
to the importance of interpersonal and spiritual leadership practice in schools? . .. Routines merely 
establish the conditions for practice in organizations—the actual practices of teaching and learning 
involve levels of agency well beyond the determining structures of routines and tools. (pp. 57-58) 
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There is little to disagree with in this conclusion: To say structures matter does not mean 
individuals do not. Nevertheless, this case study would only have strengthened its empirical 
contribution to our understanding of distributed leadership in practice by examining how routines 
and structures were established. By neglecting this conceptual realm, where micropolitics and 
leadership interact, the work may leave practitioners with the assumption that when conflict 
arises within their own setting, something is either wrong with them, their school leaders, or with 
their distribution of leadership. By leaving micropolitical conflict out of the leadership narrative, 
researchers may be encouraging leaders to consider their roles too narrowly, with potentially 
negative consequences. And as Bolman and Deal (2004) concluded, “Leaders fail when they 
take too narrow a view. Unless they can think flexibly about organizations and see them from 
multiple angles, they will be unable to deal with the full range of issues they inevitably encounter” 
(p. 433). 

The Spillane and Diamond anthology does seek to debunk one of the “myths about distributed 
leadership,” namely, that “distributed leadership is only about collaborative situations” (p.152). 
The editors argued that “a distributed perspective applies to situations where leaders have different 
or contrary goals as easily as it does to situations where leaders are striving for the same goal” 
(p. 152). In other words, the authors envision distributed leadership as a concept flexible enough to 
include conflict. Within their collection, Hallett (2007) described a school where a new principal, 
under intense pressure to raise test scores quickly, faced intense resistance from veteran staff. As 
Hallett explaind, 


The teachers, the very people on whom [the principal’s] success depended, were offended by her 
efforts to increase accountability. They refused to follow, and their own efforts to lead a movement 
against [the principal] resulted in an investigation of [the school]. The struggle to deal with all of 
these tensions was painful for everyone. (p. 102; see also Shipps & White, 2009/this issue) 


There are two relevant points to make here. First, this chapter’s inclusion in a distributed 
leadership volume indicates that the editors are aware of the important role conflict can play in 
schools, and that a study of how leadership is distributed must sometimes include examination 
of conflict. Second, Hallett’s chapter examines school micropolitics without using the language 
most familiar in the field. Although he examines coalitions, conflict of vision and goals, and fights 
for scarce resources, he does not utilize the vocabulary that, from a micropolitical perspective, 
would seem most to apply, concluding, “The leadership struggle at Costen Elementary teaches 
us that leadership is not inherent in organizational positions” (p. 102). This observation echoes 
examinations of the difference between authority and power, or between position and influence, 
found within some micropolitics writing, but it does not use those terms. Does this distinction 
matter? 

It matters in the sense that the inclusion of micropolitics into both the conceptual and method- 
ological underpinnings of their work would only seem to strengthen it, but it remains neglected 
even as an area of further research. For example, Spillane and Diamond (2007) concluded their 
volume with a call for the development of “valid and reliable ways to identify leadership and 
management practice” (p. 165), and yet the tools they evidently would like to see emerge do 
not appear to identify squarely the micropolitical processes that lie at the heart of the work of 
leadership. They acknowledged that after 9 years of field research for this project, 
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leadership and management practice is not confined to structured and patterned time slots; it happens 
before and after school, on weekends, and during school hours both in formally designated routines 
such as grade-level meetings and during informal interactions over lunch or in exchanges snatched 
between classes. (p. 165) 


But because their focus is solely on leadership and management, as if these were politically 
neutral processes, and because they do not foreground what constitutes leadership and manage- 
ment or what provides the setting for their enactment—the political realm—the instruments they 
call for seem to leave out a crucial piece of practice, which they have set as their goal. 

Larry Cuban (1988) explained the political nature of teachers’ and administrators’ work in 
schools this way: 


To the degree that teachers, for example, use their legitimate authority to allocate scarce resources to 
children, govern minors through a series of techniques, negotiate order, and bargain with members of 
the class, teachers act politically. Determining who gets what, when, and under what circumstances to 
achieve desirable ends—a classic formulation of political behavior—occurs in classrooms, schools, 
and districts. Although the settings differ for a kindergarten teacher and a high school principal, the 
core roles are the same. (p. xix) 


A micropolitical lens need not be the only one utilized to shed light on school microprocesses. 
Not only is a descriptive analysis of how leadership is distributed by definition a study of “who 
gets what, when, and under what circumstances,” but also the current educational environment 
emphasizes leadership as one of the best policy levers to pull to “achieve desirable ends.” The 
lack of inclusion of a micropolitical perspective within the distributed leadership literature seems 
a missed opportunity both conceptually and strategically. 

I make this critique cognizant of the fact that within some of my own published research there 
are missed opportunities for the examination of professional conflict and collegiality. In a study 
that focused on alternative approaches to formal leadership arrangements in schools (Grubb & 
Flessa, 2006), we said little about micropolitics although potentially interesting insights could 
have been generated. Our research examined the ways that different schools had experimented 
with leadership arrangements alternative to the traditional principal/assistant principal hierarchy, 
and we uncovered a wide variety of alternative leadership approaches: dual principalships, co- 
principalships, rotating principalships, and a school with no principal at all. Our purpose was to 
understand the trade-offs involved in pursuing the different models: What did they cost? Were 
they sustainable? Did stakeholders at the schools like them? We reached some general conclu- 
sions and highlighted particular sticking points at specific sites. But we did not write an article 
as fully attentive to the micropolitical processes at individual sites as we could have. 

The largest micropolitical lesson learned in our research is that alternatives that diverge most 
from the principal/assistant principal model conflict most forcefully with the grammar of school- 
ing (Tyack & Cuban, 1995, p. 9). Such alternatives therefore require constant justification both 
inside and outside the school. All school community stakeholders—teachers, parents, community 
agencies, and students—have experience with more typical approaches. When leadership is dis- 
tributed in ways that are in tension with taken-for-granted scripts of school administration, when 
the interpersonal and professional relationships of adults change, it would be quite surprising if 
no conflict resulted. 
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Yet our study did not examine how micropolitical conflicts shifted and changed over time. 
Although the data were rich in their descriptions of the strengths and weaknesses of alternative 
approaches, we did not gather observational or longitudinal data that would enable us to describe 
the ways that stakeholders engaged micropolitics at different decision points. Our published 
work had more to say about policy implications of different alternative responses than it did 
about micropolitics between and among individuals. 


Distributed Leadership Research That Bridges the Two Perspectives 


There are a small number of examples of work that make explicit connections between the dis- 
tributed leadership and micropolitics literatures. For example, Johnson’s (2004) study from South 
Australia found that distributed leadership was itself a micropolitical strategy for implementing 
school reform projects. Calling it part of the “‘positive politics’ of negotiation, collaboration and 
conflict resolution to address issues of local concern in schools, rather than the ‘controlling pol- 
itics’ of new managerialism” (p. 267), Johnson observed that “school leaders candidly admitted 
using micropolitical strategies and tactics to select key staff to join these teams and to set the tone 
for their operation” (p. 274). Although not utilizing the term distributed leadership, West (1999) 
observed that “more democratic [organizational structures] tends to mean with rather more scope 
for micropolitical influences over directions and decisions” (p. 190). This observation led him to 
conclude that “it is important therefore that school leaders have deliberate strategies that they can 
draw on to try to minimize the micro-political influence of informal groups on school direction 
and policies” (p. 194). Although this conclusion reasserts a view of conflict and micropolitics as 
inherently negative and of leadership as its inherently positive counterbalance—conclusions that 
are potentially limiting—West has nevertheless pointed out some ways that school leadership and 
micropolitics are intertwined. 

One recent example of research that stands out from the rest in its attempt to integrate distributed 
leadership and micropolitical perspectives is that of Storey (2004). Storey used case study data 
from England to deliver a critique of previous research in distributed leadership, taking as her 
starting point skepticism of the “mainly enthusiastic literature on leadership” (p. 249). Her article 
also succinctly lays out a handful of possible explanations for the popularity and prevalence of 
distributed leadership within the larger leadership discourse. For example, she argued that “the 
new emerging wisdom is that no single head or sole leader can handle the enormous complexity 
[of organizational leadership], hence the need for, and the attraction of, the concept of “distributed 
leadership” (p. 252). Storey also asked questions not commonly observed within this body of 
literature. For example, she sought to understand “what dynamics are unleashed when leadership 
is exerted at multiple points in a school organization?” (p. 254). 

At the heart of Storey’s analysis is a causal argument. Her analysis of case study data “reveals 
and explains how distributed leadership resulted in tension and conflicts between competing 
leaders” (p. 250), particularly between a department head and a school head. In other words, 
hers is a paper that seeks to show distributed leadership as cause, conflict as effect. Storey (2004) 
concluded that 


the experiment in distributed leadership which is examined in this case surfaced fundamental tensions 
between competing leaders, namely the head teacher and significant others occupying positions as key 
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subject leaders. These multiple leaders came increasingly into conflict as their competing visions, 
models and ideas of “success,” “good practice,” appropriate performance measurement at whole- 
school, departmental, and individual levels, became increasingly evident. (p. 253) 


Storey’s study is partly a story of the pressures exerted on school leaders within an account- 
ability environment that expects improvement quickly. It shows that conflicts arose around the 
content of leadership decisions, the appropriate realms for distributed decision-making, and the 
symbolic and substantive significance of different interpersonal approaches. As Storey (2004) put 
it, “Three issues emerged as critical: conflicting priorities, targets and time-frames; boundaries 
of responsibility; and competing leadership styles” (p. 257). 

Storey’s work is at least as much about the conflict between the rhetoric of distributed 
leadership and the lived realities of leadership delegation at the school site as it is about distributed 
leadership per se. Storey’s explanation of the different leadership approaches of school head and 
the head of the science faculty, in the final analysis, relies less on the conceptual foundations of 
distributed leadership than it does upon a traditional, rational examination of how organizational 
incentives shape the approach that different policy actors take: “The different foci in this case 
were not primarily the result of different leadership styles or personality types .... Rather, they 
reflected the differences in the pressures faced by these actors” (p. 262). Put another way (and 
borrowing phrasing from scholars of decision making), where school leaders stand on certain 
issues will depend on where in an organizational chart they sit. 

With her exploration of the distribution of both leadership and conflict, Storey’s work does 
something unusual, which is why it merited lengthy consideration here. Her work links the 
distributed leadership literature and the literature on educational micropolitics. Furthermore, 
her study, which presents a school’s struggles, not a smooth story of inevitable one-directional 
development and progress, will likely be quite familiar to educators or others who would turn 
to the distributed leadership literature in a search for greater understanding of school change. 
Although work such as Storey’s remains rare, it suggests the possibilities for developing both 
practical and theoretical insights latent in connecting these two strands of research. 


THE LITERATURE ON EDUCATIONAL MICROPOLITICS 


There are numerous studies that examine within-school politics, from a variety of perspectives. 
Scholars of this field note that although the amount of research in micropolitics has, naturally, 
grown over time, the field’s “conceptual boundaries and distinctive features remain elusive and 
contested” (Malen & Cochran, 2008, p. 148). Some studies examine the ways that leadership 
approaches or strategies encourage teachers’ empowerment (see Blase & Blase, 1997). Other 
studies examine the ways in which the implementation of new governance structures has an impact 
on teachers’ sense of collective responsibility or positive school culture (Bryk, Sebring, Kerbow, 
Rollow, & Easton, 1998; Lee & Smith, 2001). Some works explore the ways that race (McKenzie 
& Scheurich, 2004) and gender (Datnow, 2001) shape professional relationships among educators 
and between educators and students. There is no lack of articles that set out to make a point similar 
to one embedded within this paper, namely, that leadership and administration literatures neglect 
micropolitics (see especially Malen, 1995; Mawhinney, 1999). Others have made the point that 
more attention to micropolitics might assist practicing administrators more than the normative 
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idealized theorizing that, in a caricature, represents leadership and administration writing. What 
does the field stand to gain or lose in this way of framing a conversation about leadership? The 
purpose of this section is to highlight research in educational micropolitics that includes research 
questions or findings that potentially could connect to research in distributed leadership. 

Blase and Blase (1997) disagreed with a conceptualization of school micropolitics that empha- 
sizes conflict to the exclusion of consensus. In work that examines the micropolitical strategies 
school leaders use to achieve school goals, they assert that “micro-politics deals with the realm of 
cooperative (i.e., collaborative, collegial, consensual, democratic) as well as conflictive forms of 
interaction in organizational settings” (p. 138). In their work they seek to complicate the fact that 
“historically, the study of micropolitics has focused on conflictive, negative, and dysfunctional 
relations between and among individuals and groups in organizational settings” (p. 160). 

In a comprehensive review of the micropolitics literature, Malen (1995) concluded that “the 
politics of schools has received more attention than the politics in schools” (p. 148). She finishes 
her review with both methodological and theoretical recommendations for the field. 


While there are exceptions, the tendency for processes to maintain familiar power configurations is 
pronounced. Our capacity to account for these dynamics is constrained by the lack of longitudinal 
and comparative data. ... More robust designs that probe actor relations, the conditions that produce, 
perpetuate, or precipitate shifts in patterns of politics and the consequences of these styles of play for 
the distribution valued outcomes would bolster our ability to interpret the politics in schools. (p. 160) 


What is interesting about these recommendations is that distributed leadership research seems 
to have taken up one of Malen’s calls for action but not the other. Inarguably, distributed leadership 
research has sought to examine the ways that leadership is enacted outside “formal decision areas” 
and “familiar power configurations.” At the same time, it would be difficult to identify a consistent 
line of research within the field of distributed leadership that has attended to “shifts in patterns 
of politics” over time. 

The literature on teachers’ work also provides some insight regarding the ways that school 
micropolitics and school leadership are intertwined. Some of this work examines the meaning 
and purpose of conflict. For example, in her case studies of middle schools, Achinstein (2002) 
concluded that 


conflicts give teachers the opportunity to look at schools as they are and decide what they can become. 
Conflict, it turns out, offers a context for inquiry, organizational learning, and change. As colleagues 
air differences, build understanding across perspectives, and seek changes enhanced by divergent 
thinking, conflict becomes constructive for the community and school. (pp. 2-3) 


Much of the literature on micropolitics views micropolitical conflicts within schools as patholo- 
gies to be healed or obstacles to be overcome. Mawhinney (1999) summarized this perspective 
by stating that micropolitics “swirl in and around schools, often constraining the systemic efforts 
at changes” (p. 160) and that “members of the school community must find ways of resolving 
conflict which ensure that decisions made command enough respect that they are widely ac- 
cepted” (p. 160). Although likely not its intention, this way of characterizing politics suggests 
not only that it is fundamentally obstructionist but also that it is possible within schools to arrive 
at a time and place where politics and conflict are not present. Because the leadership literature 
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often takes as its starting point a normative notion that “leadership” is about progress—“We 
refer to leadership as change-related functions such as setting a vision and goals for the school 
and motivating stakeholders to move towards their achievement”—and because micropolitics is 
commonly viewed as that which gets in the way of progress, it is not surprising that the literatures 
would not overlap, even though the experiences of leadership and politics in schools do (Heck & 
Hallinger, 2005, p. 240, note 2). 

Although not described as an examination of “distributed leadership,” Coburn’s 2006 article 
does focus on what she calls “authority relations” and “the micro-processes of problem framing 
during implementation” (p. 347). Coburn identified a variety of gaps in the literature on policy 
implementation that have significance for the study of school leadership. For example, she noted 
that policy studies often examine how leaders’ understandings and interpretations of policy have an 
impact on implementation. “But this line of work has not addressed questions of problem framing 
and, as such, has tended to ignore this crucial aspect of the policy implementation process” (p. 
344). There are parallels in Coburn’s assessment of the literature to some of the critiques of the 
leadership literature, namely, that rarely do advocates of new or better “leadership” fully articulate 
an answer to the question of leadership: For what? Or to what end? 

The strength of Coburn’s article may provide one answer to why there is relatively little ex- 
amination of micropolitics within the distributed leadership literature. She relied on ethnographic 
methods and in-depth analysis of one school. This approach enabled her to observe micropolitical 
orientations and negotiations over time, and her article contains many chronological descriptors, 
for example, noting when previously nascent issued “bubbled up” (p. 353). Evidence of when, 
how, and with what results different micropolitical processes emerged was possible because of 
the single-case nature of the research design; a multicase study would have necessarily sacrificed 
this degree of detail. With such rich data, Coburn is able to pay “special attention to the role of 
authority in the interactive process of making meaning in schools” (p. 348). By studying the give 
and take between and among school community members as instructional reforms are attempted, 
she highlights the way that conflicts and micropolitics shift and change over time. Snapshots and 
interview studies are unlikely to produce similar data sets. 

Coburn’s conclusions should resonate with scholars of leadership for a variety of reasons. 
Neither simply a study of policy implementation nor of teachers’ professional community, her 
work “provided evidence that all voices are not equal in the social negotiation of meaning. Rather, 
the way in which individuals jointly construct their understandings of policy is shaped by and in 
turn shapes authority relations” (p. 373). Coburn’s study of micropolitics revealed not only that 
authority and resources—leadership—are distributed within schools, but also why and how. An 
examination of leadership without an examination of the politics of issue framing would have 
been unlikely to generate this deeper understanding of policy implementation. 


EXPLAINING THE DISTANCE BETWEEN THE TWO AREAS OF STUDY 


In an attempt to understand how two bodies of literature with common interests have developed 
in ways that are more divergent than convergent, I turn to three potential explanations. In the three 
sections that follow I propose that part of the explanation derives from the fact that at their core, 
the research problems are framed differently, that the research questions are pursued differently, 
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TABLE 1 
Comparison of Distributed Leadership and Micropolitics Research in Education 


pi ek hE eee 


Distributed Leadership Micropolitics 
ged tects | eri Joye Le bei ee al a oO SS 
Typical narrative that Distribution of influence among leaders Schools are shaped through dispute 
frames the literature shapes school culture and outcomes _ Emphasis on conflict and problems 


Emphasis in normative literature on 
progress and solutions 


Typical methodological Surveys (self-reports) of administrators In-depth qualitative methods including 
approach and other school leaders. Potentially ethnography. Particular and small 
large scale. scale. 

Case studies relying upon observations Case studies relying upon observations 
of public meetings and interviews with of public meetings and interviews with 
key subjects key subjects 

Marketplace for this School leaders and policymakers Researchers 
research Researchers 


a 


and that the marketplace for educational research creates more attractive incentives for separate 
than interdisciplinary inquiry because researchers are rewarded for differentiation (see Table 1). 


Problem Framing 


Part of the explanation for the distance between these two potentially convergent bodies of 
literature is found in how the research phenomena themselves are framed. Methodological and 
conceptual decisions define those aspects of school life considered pertinent and significant and 
those which are not. One explanation for the distance between micropolitics and distributed 
leadership research is that although potentially compatible bodies of work, the issues that they 
consider and the questions they pursue have been constructed separately. 

Many observers of policy and politics describe and analyze the ways that issue framing in- 
fluences decision making. For example, Cuban (2001) used the words “framing” and “defining” 
interchangeably (p. 4) when discussing how to solve problems and manage dilemmas. Frame anal- 
ysis represents its own methodological approach to understanding policy issues. Anderson and 
Rodway (2008); Coburn (2006); and Creed, Langstraat, and Scully (2002) provided overviews 
of this approach and demonstrated how it can be brought to bear on particular issues. Bolman 
and Deal (2004) offered one of the most accessible overviews of how to use different concep- 
tual frames to understand managerial and leadership issues within organizations; their work is 
influential in educational leadership circles because it is frequently taught in educational adminis- 
trator preparation and development programs. Bolman and Deal articulated four alternatives—the 
structural frame, the human resource frame, the political frame, and the symbolic frame—and 
suggested that although some issues may be best understood from only one frame, leadership 
requires agility in using all four lenses. The distributed leadership literature’s attention to deci- 
sion making roles and responsibilities and the impact on a school’s sense of collective endeavor 
would suggest that it views organizations fundamentally from the structural and human resource 
frames; this emphasis may lead it to underemphasize inquiry into both the political and symbolic 
substance of leadership. 
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Perhaps the best known writing on the framing of national political issues comes from cognitive 
linguist George Lakoff (2004, 2006). His work emphasizes that framing political issues is more 
than simply spinning them with different shorthand; framing is the structural way that different 
moral and conceptual priorities are articulated and the way that momentum is organized to achieve 
them. Although his work examines macrolevel political framing, there are lessons for the topic 
under consideration in this article. Micropolitics and leadership distribution are framed within 
the research and professional literatures as occupying entirely different realms of inquiry and, it 
seems, different sets of skills and interests. To study the work of leadership and micropolitical 
skill so separately seems to make a larger point about how we conceptualize both. Is this framing 
implicitly suggesting that leadership is best understood by studying it unmuddied by politics and 
that micropolitics make leadership less important? 

Although Deborah Stone (2002) examined political decision making and not educational mi- 
cropolitics or educational leadership per se, her conclusions about the ways that policy arguments 
are framed also have implications for how within our fields of research we choose to organize 
our own knowledge. She said, “Whether they are conceptual, physical, or political, boundaries 
are border wars waiting to happen. At every boundary, there is a dilemma of classification: who 
or what belongs on each side?” (p. 382). By deciding to include management in their theorizing 
about distributed leadership but not politics, Spillane and Diamond (2007), for example, classified 
politics as outside the realm of investigation. That these boundaries are created and re-created 
rather than natural or inherent seems clear from an examination of the literature. Less clear is what 
purposes are served by these distinctions. Perhaps, as Stone suggested, this categorization is evi- 
dence of an argument of what matters most. She said, “Policy argument takes the general form of 
claiming that something should be included in or excluded from a category” (Stone, 2002, p. 380). 

Ogawa (2005) argued that “different conceptions of organizations should produce correspond- 
ingly different conceptions of leadership” (p. 93). It would follow that a conception of distributed 
leadership that incorporated the micropolitical realities of the school would have at its heart a 
conception of the organization as a place where outcomes are largely the result of individuals and 
coalitions pursuing their own interests, resulting in fights over real and symbolic resources, in a 
school’s goals and vision as well as its curriculum, schedule, and markers of status. The fact that 
most of the distributed leadership research does not include a micropolitics dimension implies 
that the underlying conception of schools as organizations is one where dynamics other than 
micropolitics are most central. It is possible that a rational, potentially closed systems perspective 
(see Scott, 1987) informs distributed leadership research, where organizations are understood 
primarily as technical, managerial domains. 


Methodology 


Another explanation for the distance between distributed leadership and micropolitics research 
may be found in the different methodological demands in each area of study. As mentioned earlier, 
micropolitics studies tend to be extensive, expensive, small-scale ethnographic investigations 
whereas some of the distributed leadership literature pursues larger samples of schools. 

Sample size aside, methods questions are suggestive in other ways. For example, establishing 
the contribution that her research regarding gender politics makes to the literature on school 
change, Datnow (2001) observed that “what are common to definitions of micro-politics are the 
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elements of subversiveness and seediness. It is therefore not surprising that most studies of school 
change do not address micro-politics, particularly because of the research relationships and ethics 
that might be placed at risk by it” (p. 132). Concluding that “as educators and researchers we 
need to learn more about how teachers with competing interests and ideologies gain consensus 
on the means and the substance of school change” (p. 151), Datnow advocates for school-based 
inquiry that examines politics more explicitly. Similarly, Malen and Cochran (2008) asserted that 


given the prevalence of more covert and murky manifestations of power, scholars who focus on the 
micro-politics of schools may have to make comparable investments [a great deal of time at the site 
of study] to display, more explicitly and systematically, how all the faces of power might be manifest 
in schools. (p. 169) 


These recommendations are consistent with the arguments made in this review. What remains 
unclear is just how likely such research is to occur in a research policy environment that some crit- 
ics argue is increasingly working at cross purposes to good ethnographic practice (see Haggerty, 
2004; van den Hoonaard, 2003, for critique of research ethics review boards). 

From an historical perspective, the methodological approaches that separate distributed lead- 
ership and micropolitics may be traceable to the Progressive Era impulse to take politics from 
schooling and to professionalize and rationalize school management (Tyack & Hansot, 1982). 
School site leadership issues have commonly been examined as technical and managerial issues 
with methodological approaches borrowed from administrative science. 


The Marketplace for Educational Research 


Economists explain the state of education research by examining its market; they are concerned 
with how the realities of supply and demand create incentives that make certain kinds of research 
production more likely than others. Goldhaber and Brewer (2008) found that “the market for 
education research may not function efficiently, in the sense that not enough of the research that 
is needed gets done and too much of what is not needed is produced” (p. 201). They argued that 
within universities, departmental autonomy (they use the word “silos”) and the correspondingly 
separate incentive systems codify boundaries between different areas of research. Subsequently, 


there is often relatively little interaction and collaboration across these boundaries. This autonomy has 
important implications; for example, it reinforces the production of work based on one disciplinary- 
based view of the world, and it minimizes the sharing of methods and new developments and 
institution-level scrutiny of the quality of research. (p. 207) 


Furthermore, they concluded, “it is not uncommon for researchers in one discipline studying a 
particular issue to totally ignore a robust literature on that issue outside their discipline” (p. 210). 
Although Goldhaber and Brewer’s diagnoses are compelling, they do not perfectly match 
the case of educational micropolitics and distributed leadership. Departments of educational 
administration house many of the scholars who have written works cited previously; this silo 
includes the surprisingly separate domains of micropolitics and leadership, suggesting that the 
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incentive structures that Goldhaber and Brewer identify notwithstanding, other factors may shape 
the development of distributed leadership literature. 

Like many others, Goldhaber and Brewer are skeptical of the quality of educational research 
produced by professors working inside schools of education (for critiques of education schools 
from within, see Labaree, 2004; Lagemann, 2000; Levine, 2007). They advocated for “longer- 
term, more rigorously designed studies” (p. 209) but they also recognized that “studying schools 
and school districts requires access” (p. 209). 


Cordial relationships must be established, nurtured, and preserved. Although this may not mean 
researchers pull punches with respect to their findings, it does suggest that there is an incentive to 
study issues that are not likely to embarrass the key constituencies and undermine the relations that 
will allow for future studies. (p. 209) 


By drawing attention to the potential “to embarrass the key constituencies,’ Glodhaber and 
Brewer have made observations parallel to those of Datnow (2001), who suggested that conducting 
micropolitical research in schools has the potential to place participants at risk. Both observations 
would explain a general reluctance to examine micropolitics. Whether this disincentive also 
explains the disjuncture between distributed leadership and micropolitical research is not clear. 

Micropolitics research and distributed leadership research are dissimilar in their popularity: 
Distributed leadership is much more likely to find its way into policy and professional literatures 
than writing on micropolitics. Louis (2005) provided one possible explanation for the different 
levels of popularity that these two fields have achieved with policymakers. In her analysis of 
the relationship between policy researchers and policymakers, she described the ways that both 
intentional and inadvertent politics of production affect what knowledge is taken up in policy 
discussions (p. 233). She also explained the power that issue networks have on the distribution 
of ideas. “It is the existence of overlapping networks that may account for the rapid diffusion of 
some research ideas. Other equally valid research may languish because it cannot find its way into 
a network” (p. 228). Thanks to the current policy emphasis on leadership, distributed leadership 
literature can be slotted quickly into several networks more seamlessly than can micropolitics 
research. 


CONCLUSION 


The divergence between distributed leadership and micropolitics research is noteworthy because 
the practice of school leadership, in both its formal and informal manifestations, includes admin- 
istration, management, and micropolitics. Therefore, to neglect micropolitics is to neglect one of 
the themes that the field itself has identified as centrally significant. What does this divergence 
explain about the new politics of educational leadership? 

The current educational policy environment can be characterized by two trends that often 
work in complementary ways. On one hand, policymakers have demonstrated an emphasis on 
centrally designed and coordinated mandates with explicit benchmarks set at the state or federal 
level (see Levin, 2008). On the other, both researchers and policymakers have paid increasing 
attention to “leadership” at the school level as the policy lever best positioned to remedy school 
underperformance; entrepreneurial, “no excuses” principals are central to this vision of leadership 
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(see Carter, 1999; Leader, 2008; also see Copland, 2001, for a critique). In this era, school leaders’ 
autonomy is potentially constrained to the observable implementation of mandated curricula or 
specific class sizes, or the achievement of certain success rates on exit exams or school scores 
awarded under inspection regimes, or the display of particular wall charts under a certain approach 
to literacy. Given this context, although principals’ midlevel management activities are central 
to school-based change, it is not entirely a surprise that research on leadership at the school 
site would describe the ways conflicts could be eliminated or prevented from standing in the 
way of effective implementation. The split between the micropolitics and distributed leadership 
literatures is an artifact of the new politics of educational leadership. When policy directions for 
schools are set far away and further up the hierarchical chain of command, local micropolitics 
can be seen solely as managerial obstacles to be overcome. 

Schools will not occupy this policy moment of centralized coordination forever and it will be 
important for our understanding both of leadership and of this historical moment to look back 
in a few years to see if distributed leadership is an expression of a particular kind of school 
administration with its roots in the accountability movement, or whether it has theoretical legs 
that will help understand leadership in more variable policy situations. Malen and Cochran (2008) 
similarly raised a call to pay attention to the ways that micropolitics are shaped in this particular 
policy moment. 


To be sure, both longitudinal and comparative studies are required to gauge more precisely how 
actions taken at higher levels of the systems are permeating, if not dominating the micropolitics of 
schools. But the available research suggests that macro-forces may be controlling the agenda, limiting 
the latitude, restricting the scope of influence and otherwise circumscribing the power of site actors. 
If these findings hold, the macroforces in the policy environment may be among the most critical 
factors affecting the micropolitics in schools. (p. 168) 


The problem with ignoring micropolitics in the study of school leadership is that the vision 
of schooling that is left behind is one that is not only overly simplified, but it also reinforces 
an idea that conflict within schools is a pathology best avoided. For those who consider schools 
essentially neutral or positive places where micropolitics distracts from schools’ more important 
functions, this stance makes sense. But for those who see schools as always shaped by conflict 
over goals and purposes, attending to micropolitics means that important concerns about school 
unfairness or inequity, ignored in the standard operating procedures from day to day, might come 
to light (see, e.g., Singleton & Linton, 2005). According to this second perspective, the skill 
school leaders need is not the perfect strategy to engage in the impossible struggle to eliminate 
micropolitics but instead the skill of inquiry to look for it in every instance, to examine who 
gains and who loses under important school decisions, and to develop a better understanding of 
the concerns that motivate people to join together in support or opposition of school initiatives. 
Practicing a skill of inquiry means neither relinquishing overarching commitments nor bending 
to whichever group’s voice is strongest. Instead, it means viewing politics as inevitable and 
using the always-latent conflicts as generative of information about what matters most to school 
stakeholders. 

Not all conflict is productive, and neither will attending to all micropolitical dynamics within 
a school lead to positive results. As Marshall and Scribner (1991) explained, 
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On one hand, micropolitical analysis can provide eminently practical insights for practitioners. 
Educators can use micropolitical skills to plan alterations in resources and manipulation of symbols 
to reduce inequities and to increase the power and voice of previously powerless groups. . .. However, 
these same skills could be used as easily to advance less socially desirable political agendas. (p. 352) 


Acknowledging political dynamics and pursuing them intentionally requires articulating the 
values that drive and organize leadership decisions. It may be exactly this fact that most stands 
in the way of a merging of distributed leadership and micropolitical analyses, for the distributed 
leadership literature, like much of the leadership literature in general, is far more likely to take up 
technical issues of school organization than it is to examine what significance or relevance those 
issues have (see Noguera, 2006). Micropolitical conflicts provide information. Sometimes the 
information tells us that people are shortsighted or selfish or irrationally stubborn. But sometimes 
the information shows us what adults in schools value in their professional practice and what 
families aspire to for their children. Understanding those values and attempting to serve them are 
leadership responsibilities, and research that seeks to inform leadership practice should examine 
them. 
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How do principals adjust to new standards and measurement rulers of school performance when 
they are coupled with high-stakes consequences? Adopting a contingent view of leadership and 
a multiple accountabilities framework, this qualitative study explores how high school principals 
enacted their accountability in New York City, by comparing two periods: just before such policy 
changes took effect and just after. We observed large differences in principals’ attention to external 
political, bureaucratic, and market accountabilities; how much accountability conflict or alignment 
was experienced; and how principals’ sense of professionalism was constructed. These differences 
may portend a new politics of the principalship. 


Principals have a power advantage over other school actors. Betty Malen and Melissa Cochran 
(2008) argued that they typically use their influence to protect established interests and deflect 
criticism by “avoiding, suppressing and containing conflict” (p. 168). Yet recent policy shifts 
may be narrowing the scope of principal influence or providing opportunities for principals to 
orchestrate conflict rather than suppress it. 

One arena of policy change is accountability. Principal success or failure is increasingly 
evaluated from the perspective of reformers who aim to alter their behavior as a means of 
achieving targeted educational improvements (Wong & Nicotera, 2007). Nowhere is this more 
evident than in New York City. Each high school principal’s performance rating depends upon the 
school’s yearly grade, calculated as a function of its attendance, testing, and graduation ranking 
when compared to “peer” schools. The school grade either adds points to the principal’s perfor- 
mance rating or starts her at a deficit. Poorly rated principals can be removed in as little as two 
years; highly rated ones can earn significant bonuses (New York City Department of Education, 
2008a). 
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Despite the strong incentives in such evaluation systems, their operation is neither mechanistic 
nor uncomplicated (Bizar & Barr, 2001; Mulford, 2008; O’Day, 2002). The politics of school 
accountability are so complex and uncertain that tightly linked systems of behavior, rewards, and 
punishment may serve to relegitimize the public schools even if they do not change the way leaders 
think and act (Malen & King-Rice, 2009). To explore how accountability affects leaders’ behavior, 
we examine these politics from the inside out, that is, from the principal’s perspective. We ask, 
How do New York City high school principals construct their accountability environments? How 
do the types of schools they lead influence their experience of accountability? What changes in 
the perceived accountability environment when consequences are stiffened? How might this alter 
the politics of the principalship? 

To study these interrelated issues we interviewed two samples of high school principals in New 
York City, 3 years apart. We chose high schools because recent research suggests that their sense 
of efficacy operates differently from that of elementary school principals’ (Leithwood & Jantzi, 
2008, p. 524), teacher cultures differ greatly between elementary and high schools (Walstrom & 
Louis, 2008), and because high school principals perceive significantly more job-related problems 
than do elementary or middle school principals (Gates, Ringel, Santibanez, Ross, & Chung, 2003; 
Teddlie, Stringfield, & Reynolds, 2000). Moreover, in New York City they confront performance 
targets quite different from those designed for elementary school principals. 

Our first interviews took place in 2004-05, 2 years after mayor Michael Bloomberg took 
control of the schools in 2002, closed 36 subdistrict offices, and abolished all school boards. 
Bloomberg also mandated a common curriculum for reading and math; reorganized the oper- 
ational bureaucracy; and created 10 new instructional units, called regions, made up of 100 to 
150 schools each (Fruchter, 2008; Viteritti, 2009). Under these centralizing governance reforms, 
principals were exhorted to take responsibility for the performance of their schools, but the city’s 
performance-tracking mechanisms were still in development. State Report Cards—relying on test 
scores from the Regent’s Examinations that all New York State students must pass to graduate— 
were the primary means of assessing school performance.’ There were no special consequences 
for principals unless a school was judged so poor it must be shut down by the state. 

The second round of interviews took place in the school year 2007-08, just after the Bloomberg 
administration had implemented three new school performance measures and tied them to prin- 
cipals’ performance ratings. After 2006, the city mandated three district-developed measures of 
school performance. Each high school’s Progress Report (PR) weighs several types of perfor- 
mance data: Regent’s exams (30%), the progress of the high school in “moving” students to 
higher levels of performance and on-time graduation (55%), and attendance combined with the 
results of surveys (15%).* Learning Environment Surveys (LES) assess the satisfaction levels of 
three constituencies—parents, teachers, and students—on school safety and respect, academic 
expectations, engagement, and communication. PRs also identify whether the school received 
“extra credit” for unpredicted performance gains among English language learners, special edu- 
cation, and/or its lowest scoring students. Beginning in 2006—07 these measures were summed 
into a single score and transformed into a school grade, A-F. School Quality Reviews (SQRs) 


' As of 2007-08, to receive a high school diploma all students were required to pass five Regent’s exams, one each in 
English, math, science, global studies, and American history. 
2 As of 2008-09, these relative weights shifted to 25%, 60%, and 15%, respectively. 
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resulted from 2-day inspection visits conducted yearly by contracted educational consultants. 
SQR inspectors graded schools as “Undeveloped,” “Proficient,” or “Well-Developed.”? 

Since 2007, principals have been graded on a newly revamped Professional Personnel Review 
that relies on the PR, the SQR, the Learning Environment Surveys, and up to five goals set by 
the principal. Principals are told they have 2 years to meet.district-set performance goals or risk 
losing their jobs. Principals are also eligible to receive bonuses of up to $25,000 if their PR scores 
are in the top 20% of the city. In fall of 2008, three high school principals received this top prize, 
and another 30 received a lesser bonus (Hernandez, 2008b).* 

We did not anticipate such dramatic changes when the first interviews were undertaken. By 
2007, however, we realized our common research protocol made it possible to reanalyze our 
data for insights into principals’ actions in the face of extreme shifts in accountability policy. 
Notwithstanding our inability to conduct formal pre—postanalysis because of differences in our 
two sampling frames, our findings are strongly suggestive. If confirmed, they may portend a new 
politics of the principalship. 


FRAMING ACCOUNTABILITY AND LEADERSHIP 


Accountability consists of demands—from oneself or legitimate others—for a demonstration 
of performance claims. For example, politicians manifest accountability by listening to and 
addressing the needs of their constituents. Managerial employees show their bosses that they 
have met established targets, and are using preferred means to do so. Market-based entrepreneurs 
please their customers, or risk losing them to another provider. Lawyers demonstrate that they 
follow professionally accepted practice or risk ostracism and censure, while religious leaders live 
in conformance with the teachings of their faith. 

School leaders face all of these demands. Principals show they listen to and address the prob- 
lems of parents and community members by negotiating conflict or by mobilizing support in 
anticipation of their demands (Malen & Cochran, 2008). Principals manage the gaps between 
external standards of school performance and current performance—test targets, but also proce- 
dural requirements—by managing resources and coordinating expectations so that the standards 
become common goals (Carnoy, Elmore, & Siskin, 2003). In response to market pressures, prin- 
cipals compete among themselves for excellent students, teachers, and other scarce resources. 
When asked to demonstrate their professionalism in curricular and instructional leadership, they 
are called upon to coach and mentor their subordinates (Stein & Nelson, 2003). Like religious 
leaders, school principals use moral persuasion and ethical reasoning to justify their decisions 
(Sergiovanni, 1992). All these accountability claims have been previously identified in typologies 
(Adams & Kirst, 1999; Darling-Hammond, 1989), and by factor analysis that identifies multiple 
components of successful principal decision making (Leithwood & Jantzi, 2008; Mulford, 2008). 
Table 1, adapted from Firestone and Shipps (2005), outlines how each of five accountability types 
might be expected to influence principal behavior. 


3In 2007-08, SQRs were graded as “outstanding,” “well-developed,” “proficient,” “undeveloped with proficient 
features,” or “underdeveloped.” 


4gchools must volunteer to be in the bonus eligible pool, but all principals are subject to the grading sanctions. 


A NEW POLITICS OF THE PRINCIPALSHIP? 353 





TABLE 1 
Typology of School Leaders’ Accountabilities 
Type Manifestations Objectives Operative Mechanisms 
Political Citizen pressure Satisfaction Suppressing, negotiating or 


mobilizing conflict 
Bureaucratic 


Process Regulations Obedience Compliance 
Outcome Standards and Systemic alignment Managing resources and 
consequences coordinating expectations 
Market Entrepreneurialism Creativity, efficiency Competition 
Professional Practical consensus Preferred practice Coaching and mentoring 
Moral Espoused beliefs Value commitments Moral persuasion or ethical reasoning 





Note. Adapted from Firestone and Shipps (2005). 


As Table 1 indicates, reform efforts create new, overlapping, and sometimes competing ac- 
countability obligations. For example, principals are expected to achieve bureaucratically targeted 
goals first, even if this means putting off other educational objectives that teachers, parents, and 
students value (see, e.g., Linn, Baker, & Betebenner, 2002; Sleeter, 2007). Yet, in New York City 
as elsewhere, open enrollment and the creation of hundreds of small high schools has pitted prin- 
cipals against one another in competition for students whose prior performance predicts they can 
meet outcome expectations (New York City Department of Education, 2008b). At the same time, 
parents are clamoring for more voice in school disciplinary policies, criticizing school leaders 
for a lack of consultation (Commission on School Governance, 2008). Accrediting associations, 
teacher-training institutions, and teachers also do not easily forgive what they perceive to be 
nonstandard professional practice even when it is initiated in response to outcome goals. Fairness 
remains important; many urban school systems, including New York City’s, serve a highly diverse 
student body and are located in economically stratified cities. 

School principals’ accountability tasks are also situation specific. One recent study based on 
a multiple accountabilities framework concluded that variations in influence by state agencies, 
parents’ associations, teachers, and school councils on instruction and supervision decisions 
significantly impacted how principals interpreted their own power and influence. Teachers and 
parents’ associations seemed to pull principals in opposite directions; strong teacher influence 
diminished principals’ willingness (or ability) to be influenced by parent groups and vice versa 
(Marks & Nance, 2007, p. 25). 

Contingent and goal-directed leadership theories better capture our view of the principalship 
than do trait, skill, or style theories (Northouse, 2006). We adopt a contingent approach in which 
leaders perceive their responsibilities as varied depending upon the relationships they have with 
adults crucial to their organization’s performance, the structure of the educational task, and 
the positional power they have to reward or coerce desired behavior. At the same time, we 
assume principals are goal oriented; they must motivate teachers, students, and parents to achieve 
outcomes that are frequently challenging. 

In this view, principals do not have the luxury of simply responding to the loudest or even 
the most immediate accountability pressure. They also do not simply apply their acquired skills 
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or leadership styles, as much of the survey research assumes. Instead, principals construct their 
accountability environments, focusing on some concerns over others, balancing competing de- 
mands, and inevitably making choices (Firestone & Shipps, 2005). One principal may experience 
zero-sum choices between competing sets of accountability demands, even constructing her ac- 
countability environment so that two are always in existential conflict. Another principal, or the 
same principal at a different time, may experience the same accountability demands as aligned: 
several types of accountability reinforcing one another. Still others may focus on a single ac- 
countability type that has grown distinctive enough in particular school environments to crowd 
out all others (White, 2006). 

Before we analyzed our data, we developed hunches about the differences we anticipated with 
respect to the school’s adult relationships, the structure of the educational task, and the leader’s 
positional authority. We measured variations in adult relationships by the school size and the 
principal’s gender. We measured variations in the school’s educational task by the level of student 
diversity and by students’ aggregate level of pre-high school achievement. These two dimensions 
of leadership are assumed to be stable factors of influence, changing little, if at all, over the 3 
years between 2004-05 and 2007-08. 

We measured variations in the principal’s positional power by contrasting the ways they 
enacted their accountability environments in 2004-05 and 2007-08. This measure would di- 
rectly respond to the shifts in district policy that the New York City Department of Education 
(DOE) adopted but also serve as a proxy for principals’ perceived scope of influence. We at- 
tended to shifts in principals’ descriptions of the types of accountability that dominated their 
environment, how principals’ views changed about whether accountabilities were in conflict, 
aligned, or unopposed, and which specific combinations they identified as crucial to their decision 
making. 

We found strong evidence that principals reconstruct their accountability environments when 
the district changes its performance measures and consequences but were relatively unaffected 
by the adult relationship and educational task variables we used. These findings may be artifacts 
of differences in the two samples—alternative hypotheses we note throughout—but we argue that 
principals own descriptions support a policy-based explanation of the changes we found. If so, our 
results complicate an emerging argument that external demands of various kinds are dominating 
principals’ perceptions of their authority largely by reducing their perceived discretion (Malen 
& Cochran, 2008). We found a dramatic increase in the sheer numbers of external demands 
that New York City principals attend to, but the amount of conflict between these demands and 
principals’ sense of professionalism diminished over the same period. New York City principals 
now appear to acknowledge a wider array of external actors as legitimate stakeholders, and many 
are beginning to call upon them for assistance in meeting educational goals. If substantiated, our 
findings suggest a turning point in the politics of the principalship. But, before we examine the 
details, we describe our studies. 


METHODS 


The 10 principals sampled in 2004-05 were purposively selected based on their independently 
judged reputation for leading “good” schools, defined in terms of instructional competence, 
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using two of Larry Cuban’s (2003) categories—progressive and traditional.° This resulted in 
10 high schools and an initial response rate of 33%. These respondents were supplemented 
with six additional high schools using snowball methods. The 18 2007-08 principals were 
selected from a stratified random sample and offered a $50 gift certificate for their school as 
an incentive to participate. An initial response rate of 50% required that the process of random 
selection be repeated.®° New York City has a sizeable and growing number of small high schools. 
Consequently, both waves were stratified to include equal numbers of small and large high 
schools.’ 

All interviews followed the same protocol, developed by the authors and a colleague, Tarik 
Razik, in 2004 (Razik, 2006; White, 2006). The interview protocol was based on the Firestone 
and Shipps (2005) framework of multiple accountabilities and adapted critical incident technique 
(Flanagan, 1954). Critical incident technique’s open-ended questioning is not biased toward one 
or another theoretically driven view of leadership (Northouse, 2006), and it has proven useful for 
identifying actors’ perceptions of behaviors critical to their own performance in environments 
where individual discretion is high and decisions have serious consequences (Chell, 1998), like 
New York City’s high stakes accountability environment. 

Each principal was contacted to secure an appointment at the school site. Interviews were 
tape-recorded, transcribed, and coded for critical incidents.’ Permission to enter the schools 
was secured as required by college and DOE institutional review processes. All interviews were 
conducted confidentially. Names provided herein are pseudonyms and some descriptive data are 
omitted or changed to preserve anonymity. 

Our contingent and goal-directed framing of leadership requires active sense making on the 
part of school leaders. Consequently, we define an incident as a reflection of the principal’s 
perspective, not as a reliable statement of what improves performance. Often it is related as a 
story, with a beginning, middle, and end, and/or discernable causes and predicted consequences. 
This framing has intentional resonance with the narrative interests of political policy analysts, 
who judge political behavior as much by the stories that are told to justify, explain, or rationalize 
it, as by the actions that unambiguously result (Stone, 1998). For our purposes, a critical incident 
is any situation involving a principal decision that, in hindsight, seemed consequential. Leaders 
may feel certain that a particular decision on their part was self-evidently purposive, or that 
change in their working environment obviously shifted the consequences they face—believing 


5 Two additional categories of schools mentioned by Cuban —democratic and community based—were not sufficiently 
distinguishable to be useful. 

Three “slots” required only one new draw, three required a second redraw, one required three redraws. The last slot 
required six redraws. After testing the data for differences by race, gender, school performance rating, and several other 
characteristics to determine if the schools requiring a redraw were systematically different from those responding to the 
original invitation, we concluded that the most plausible reason for the redraws was simply time. The longer it took to 
schedule an interview, the less likely the principal was to accept the invitation, since the school term became increasingly 
busy with spring testing, budgetary reconciliation, and graduation. 

TIn both protocols, the population excluded schools with examination or audition entrance criteria (n = 7), schools 
for overage students, GED-only programs, schools assigned to incarcerated students and those labeled by the State of 
New York as “In Need of Improvement” for 2 years in a row. 

8The 2007-8 sample was coded using NVivo software. 
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this incident was critical—even when their intentions and/or the shifted incentives do not reliably 
have a causal relationship to outcomes.” 

The interview protocol consisted of seven open-ended questions structured to elicit incidents 
critical to the principal’s experience of accountability at the time of the interview. Principals were 
asked to provide an explanatory example, or typical story, about each type of accountability, 
although there was no limit on the number of incidents they could relate and the range proved 
wide (n = 5-17). Multiple probes ensured that critical incidents were described in detail (e.g., 
Was anyone else involved in this issue? Did you consult anyone about what to do? What was 
their response? etc.). Interviews lasted, on average, 1.5 hrs. 

We coded each critical incident for the accountability represented and coded incidents were 
the unit of analysis. Reliability was assured in two ways: The protocol was pilot tested and 
revised by the second author and a colleague in 2004. Interrater reliability was assured in 2004 
by an initial agreement between two coders of 64% with a Cohen’s kappa (taking account of 
chance) of .55, considered “moderate” by Stemler (2001). In 2008 the authors of this study 
initially achieved 75% agreement in their coding of a sample of incidents and a Cohen’s kappa 
of .41—again, “moderate” consensus. Our samples are not large enough to meet assumptions of 
statistical validity, but we expect this study will help us understand how major policy changes 
affect what principals are able and willing to attend to in their accountability environments, and 
help us understand when and how principals feel powerful. 


Characteristics of the Schools and Principals 


Table 2 summarizes some characteristics and school accountability measures that describe the two 
samples. The schools were, on average, similar in terms of student race and ethnicity, although the 
4004-05 schools were somewhat more highly diverse, whereas the 2007-08 schools evidenced 
more variation. Unsurprisingly, the peer index, the average attendance rate, the rate of freshman 
on track for graduation, and the 4-year graduation rate were all somewhat higher in the 2004— 
05 schools, which were selected based on their “good” reputation. The 2004-05 schools also 
educated far fewer English Language Learners on average than did the 2007-08 schools.!° All 
the schools had the traditional configuration of Grades 9 to 12, although three schools in the 
2007-08 group also had Grades 6 to 8. 

In one important way the principals differed: In 2004-05 their average tenure was more than 
twice as long as the average tenure of those interviewed in 2007-08. Our interviews provide a 
great deal of anecdotal evidence that there are fewer seasoned principals in the system today. In 
2007-08, some of our informants told us they intended to leave the principalship because the 
policy changes had so altered the job. But because the first sample was selected purposively and 
the second sample randomly, it may also be that the stark differences are a sampling artifact. In any 


° Some studies find low correlations between principal’s high self-ratings on these components and school outcomes. 
Leithwood and Jantz (2008) reported that in path analysis self-efficacy does not have an independent effect on student 
outcomes. 

\0Other school characteristics reflected this pattern of little or no average difference between samples except when the 
characteristic was a calculated score reflecting performance or ability to produce higher performance. 
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case, the lack of overlap precluded our analysis of principals’ responses by years of experience, 
one alternative measure of adult relationships. 


HOW DO PRINCIPALS DESCRIBE CRITICAL INCIDENTS? 


The 10 principals interviewed in 2004-05 reported a total of 96 critical incidents, while the 
eighteen principals in 2007-08 reported 241 critical incidents. As we discuss in our findings, the 
larger average number of critical incidents in the later years may suggest that stepped up sanctions 
and incentives by 2007-08 had made more types of incidents seem critical. 

Our 2004-05 informants identified 29 different combinations of accountability types and 
relationships between them, whereas in 2007-08 the range consisted of 31 distinct combinations 
of relationships of conflict and support. Every principal identified at least one example of each type 
of accountability—political, bureaucratic, professional, market, and moral—and all identified 
several instances of conflict between accountabilities that principals described as forcing a choice 
between two competing pressures. Ninety-three percent of principals experienced accountability 
incidents that were complex, in which three types of accountabilities were pulling them in different 
directions or where one kind of accountability supported another so that a principal’s decision 
making involved at least three constituencies and at least two different decision options. At the 
other end of the spectrum, 85% reported experiencing accountabilities that were aligned: two 
or more accountability types supporting the same decision. Only slightly more than half (57%) 
experienced uncontested types of accountability, with no competing pressures and no supporting 
ones either. Strikingly, all these incidents come from 2007-08. 

What follows is a summary of the top 91% of all incidents. We attenuated the full list because 
the bottom 9% represent accountability types and relationships that each occurred in less than 
one percent of incidents. Most are one-off combinations that reflected idiosyncratic coding or one 
principal’s unique experiences with accountability; the long “tails” of these two distributions. 


DISCONFIRMING EVIDENCE AND UNANTICIPATED SURPRISES 


Our initial hunches about gender, size, school diversity, and students’ prior performance were all 
based on more or less commonly accepted distinctions in the research literature. They proved, at 
best, weakly substantiated by our evidence. We briefly summarize these weak associations before 
exploring our key findings in greater detail. 

Table 3 displays the major findings of two analyses on all 28 principals interviewed. The far 
left column shows the types of accountability identified (i.e., political, bureaucratic, professional, 
market, or moral) as well as the relationships between them: Unconflicted types are reported with 
the full name, simple conflict between two types of accountability is shown by a “vs.” between 
the two, simple support is represented by a “supp” between two types of accountability, and 
complex conflict and/or support is represented by three types of accountability with a mixture 
of “vs.” and/or “supp” between them. The second two columns provide the number of incidents 
coded by type of accountability, and the proportion of all 306 incidents mentioned. The first two 
comparisons (columns 4—7) show how principals constructed their accountability environments 
when sorted by our measures of adult relationships: principal gender and school size. The last two 
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comparisons (columns 8-13) show how principals constructed their accountability environments 
when sorted by our measures of the educational task: the diversity index and the peer index. 


Gender 


Principals in both our samples were about 40% women, relatively high compared to a national 
average in 2000 of 21% women high school principals (Gates et al., 2003, p. 78). Although 
old stereotypes are shifting (Glasser & Smith, 2008; Hyde, 2005), much research on leadership 
still concludes that women, on average, lead in a more democratic or participative manner 
and engage in more contingent reward behaviors than men. They are rated less effective if 
they are directive, a putatively masculine behavior, and more effective where the organizational 
culture encourages communal interpersonal skills (Brunner & Schumaker, 1998; Hoyt, 2007). 
Consequently, we anticipated finding more female than male principals reporting moral and 
political accountabilities—which draw on empathy, collaboration, and interpersonal skills—to 
be critical to their leadership responsibilities. We also anticipated women would report less 
conflict than men. 

We were wrong. Female principals identified political accountability as an influence less often 
than male principals did, and moral accountability by itself was mentioned no more often by 
female than it was by male principals. Instead, three unanticipated differences surfaced in which 
conflict outweighed alignment. Female principals described their moral compunctions as aligned 
with their professional obligations, but they also saw their professional and political obligations 
in conflict with bureaucratic ones, and their professional and bureaucratic obligations in conflict 
with market pressures. All of these types of incidents were mentioned more than we would have 
predicted given a 40-60 gender split. However, none represented a significant difference in how 
female and male principals enacted their accountability environments. 


School Size 


We defined small high schools as those with 550 or fewer students.'’ Small high schools are 
a trend: In 1995-96, New York City had about 60, compared to 99 large ones; by 2004—05 
the number was about 160, alongside 121 large schools. Three years later, the number of small 
high schools was nearly double that of large ones: 205 to 114 (New York City Department of 
Education, 2008b). 

Intimacy is one of the intended effects of small schools. It encourages adult collaboration and 
nurtures students who need individualized support (Ancess, 2003). But the benefits of intimacy 
reliably occur only when schools adopt intensely collaborative learning structures (Lee & Smith, 
1994), hire teachers who fit these cooperative expectations (Darling-Hammond, Ancess, & Ort, 
2002), and when students experience consistency from adults (Lee & Burkham, 2003). There are 
also negative effects of small school intimacy: “Proximity may breed conflict, and very strong 
interpersonal relationships may have a negative influence on members’ decisions and behaviors” 


'1There is debate about how to define small schools. Some set the cut-point at 600, others at 500 (see Honig, 2009/this 
issue). 
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(Lee, Smerdon, Alfeld-Liro, & Brown, 2000, p. 164), problems that could be exacerbated in 
New York City where small high schools frequently share a building. Small schools are also 
many times more likely than large schools to fall into the extreme categories of high- or low-test 
score performance, or to report changes in score means and gains from year to year. Although 
some small-school principals are aware of the problem (Hernandez, 2008a), others may jump to 
“unwarranted conclusions” about their effectiveness, straining the personal relationships needed 
to sustain collaborative teaching strategies and individuation (Kane & Staiger, 2002, pp. 95, 99, 
102). 

Again, our findings were only weakly supportive of our initial hunches. We had anticipated 
that small—high school principals would report high levels of political or market accountability 
from parents and external organizations because their intimacy encouraged greater contact. We 
also expected small-school leaders to report higher levels of professional accountability because 
of collaborative decision making. We thought small-school principals would report that their 
professional, political, and market accountabilities were in conflict with their bureaucratic obli- 
gations more often than large-school principals did. Yet uncontested bureaucratic accountability 
influenced small-school principals’ decision making seven times more often than large-school 
principals’, and small-school principals reported accountability conflicts between bureaucratic 
and political accountabilities about half as often as did large-school principals. Surprisingly, only 
small-school principals mentioned uncontested moral accountability as an influence on their de- 
cisions. Overall, we are left with findings that suggest neither gender nor school size influenced 
our principals’ accountability environments in ways that the literature led us to expect. 


School Diversity 


To account for the longstanding research finding that student performance is predicated on student 
socioeconomic characteristics (Teddlie et al., 2000), we adopted as one measure of the school’s 
educational task RAND’s school diversity index: “the probability that two members [students] of 
the [school] population chosen at random will be of different subpopulations” (White, as cited in 
Gates et al., 2003, p. 75).'? In RAND’s study of national survey data, diversity values above .7 
were associated with a marked increase in principals’ perceptions of poverty-related instructional 
problems and conflict among parents and community members (p. 123). Because our sample is 
small, we categorized the RAND scale. Our diversity index scores ranged from high (.7—1.0), to 
moderate (.4—.6), or low (0-.3). 

High diversity schools do not appear to be more conflictual accountability environments for 
principals in New York City, although we anticipated that we would see results similar to the 
RAND study. Instead, we saw slightly /ess conflict in the most heterogeneous schools. Rather than 
finding higher amounts of conflict between parental demands (whether political or market) and 
principals’ professional obligations in high diversity schools, as we anticipated, they experienced 
just the opposite: Professional and political accountabilities supported one another (were aligned) 
almost 3 times as much as in lower diversity schools. More in keeping with our expectations, high- 
diversity school principals also reported that accountabilities were in three-way tension many 


12.4 value of zero indicates complete homogeneity; all students are of one racial or ethnic group. A value of one means 
complete heterogeneity; all groups are represented equally. 
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more times than did principals in lower diversity schools; principals at high-diversity schools 
were the only ones to report three-way conflict between market, professional, and bureaucratic 
accountabilities. 


Student’s Prior Performance 


The Peer Index is New York City’s quantification of the educational challenge each high school 
faces. It is a weighted average of the students’ eighth-grade English-Language Arts and math 
exam rankings that range from 1 to 4.5; the higher the number, the higher scoring the school’s 
entering students.'? Research using prior achievement as a baseline is intended to evaluate the 
relative success of interventions by controlling for what the students already know. The difference 
between where they started and where they end up is measured as a gain (see Roderick, Jacob, 
& Bryk, 2002). New York City’s DOE argues that the peer index is the best predictor of future 
student performance because it provides a baseline for judging schools on a growth model. Our 
objective in using the peer index is to acknowledge that principals in low performing schools face 
a difficult educational challenge: If students are to graduate, schools must focus on accelerating 
student performance. Consequently we anticipated that principals of low peer index schools 
would report lower levels of conflict between their internal moral and professional obligations 
and the political or market accountabilities they have to parents and other community constituents 
than did principals of high peer index schools. 

Our expectations were largely confirmed. Although principals of schools with lower achieving 
students told us about complex conflict situations more often than principals of higher achieving 
schools, they also told us that their professional and market accountabilities were aligned; out- 
distancing high-achieving schools by a factor of 2 to 1. Lower achieving school principals also 
reported much less conflict between professional and political accountabilities and, in general, 
reported fewer complex accountability conflicts. !4 

Even so, the reasons for these results may not be straightforward differences in the school’s 
educational challenge. New York City complicated the peer index as a measure of the school’s 
educational task by using it to develop a “peer group” against which the school (and consequently 
the principal) is rated for purposes of deriving a yearly school grade. Because this difference 
occurred between 2004 and 2007, it is therefore also captured in between-sample comparisons.'° 

Our findings on measures of the educational task were mixed. The school’s diversity seemed 
not to affect how principals enacted accountability in ways we predicted, and where our hunches 
about students’ prior achievement was sustained, we could attribute it to policy changes. It is to 
our most striking policy findings that we now turn. 


'3The mean peer index score for all 238 high schools that were rated in 2006-07 was 2.73, the median index score is 
2.64, the lowest score was 2.04, and the highest was 4.06. 

'4Those with lower peer index scores also reported more uncontested bureaucratic and political accountabilities than 
did their high-peer-index colleagues, likely a reflection of the sampling frame. 

'SRach school’s peer index is used to identify the 20 high schools whose incoming students score just below, and the 20 
schools whose incoming students scored just above it on the eighth-grade tests. Schools facing similar student challenge, 
it is argued, should be spurred to action by being rated against one another. 
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POLICY EFFECTS ON PRINCIPALS’ SENSE OF THEIR POSITIONAL 
AUTHORITY 


New York City’s school accountability regime is an extreme example of an incentive-laden 
“recognition and reward program” (Ladd, 1996, chap. 2). This form of outcomes-based bureau- 
cratic accountability is intended to reinforce the principal’s managerial authority (referred to as 
“empowering”) and is largely drawn from corporate experience.!® It rests on the coordinated use 
of regular public reports of school performance against targeted outcomes and personal conse- 
quences for principals. One assumption behind increased sanctions and rewards is that principals 
will figure out how to align school resources to facilitate performance beyond expectations, some- 
times referred to as “transformational leadership” (Northouse, 2006). When principals embrace 
the performance criteria of the district as official protection for decisions they want to make but 
would otherwise avoid, they become less conflict averse. If this occurs, the principal’s positional 
authority will be experienced as having been enhanced. 

We anticipated, therefore, that the adoption of personal sanctions that applied to principals in 
2007—08—but not in 2004—05—affected how they experienced their own sense of authority. We 
expected that 2007-08 principals would attend less to professional and moral accountability in 
their decision making, and more to bureaucratic, market, and professional accountabilities. We 
also expected them to identify their accountability environments as more aligned than 3 years 
earlier, when personnel and curricular decisions were independent of published performance 
ratings and personal consequences. 

Nearly all our hunches about shifts in the principal’s authority seem to be born out, and the 
numerical differences are big. Even so, we do not want to overstate these findings—they are based 
on relatively small samples and there are many potentially confounding interpretations—so we 
use the principals’ own words to clarify what we found.'” 


MAKING SENSE OF OUR FINDINGS 


Figure 1 shows that principals in 2004—05 identified professional and bureaucratic accountabilities 
as most central to their decision making, and the pattern remained 3 years later. Although 
some 2004-05 principals reported pressure from the DOE, parents, community groups, and 
competition (political and market accountabilities), moral sources of accountability were fully 
one quarter of all incidents. By 2007-08 political and market accountability had replaced some 
of the professional and moral compunction. 

Figure 2 shows another difference in principals’ enactment of their accountability environ- 
ments. Both complex conflict and simple conflict were less evident in 2007-08, replaced by a 


16New York City’s reforms were explicitly intended as an extension of Chicago’s, where the outcomes-based bureau- 
cratic accountability scheme—also implemented when a mayor took over the schools—was most strongly influenced by 
state and local corporate business associations (Shipps, 2006). 

!7Because the number of small high schools is growing, and our second sample contained some new schools, it may 
be that our small school measure also interacts with policy changes in accountability. We find this especially plausible 
since distinctions between small and large high schools showed a pattern similar to that reported in this section. 
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FIGURE 1 How did principals differ in their perceptions of which accountabilities they face? 


sharp increase in principals’ perceptions that every type of accountability influenced them directly 
and without contestation. Table 4 and the principals’ own words help us explore the details. 


Enduring Conflicts: Professionalism and Bureaucracy 


The number of incidents characterized by bureaucratic and professional conflict tops all others in 
both periods. All 28 principals told us about at least one such incident, and 21 gave us more than 
one. Consequently, we divided this huge category into several subcategories to learn more. 
Principals in 2004-05 thought of this core conflict in two ways. One was the emerging 
influence of tested outcomes competing with the pedagogical goals schools had for their students. 
Principal Pine experienced the conflict emanating from the state: “We tried to provide services 
for our students who were special ed[{ucation] students whether the services were mandated or 
not and we had a struggle with the [State Board of] Regents. ... We were basically told that we 
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FIGURE 2 How did principals differ in their views about how accountabilities interact? 
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TABLE 4 
How Have Principals’ Perceptions of Their Influence Been Altered by Policy? 





Influence of Policy Changes 


2004-05 2007-08 





INCIDENT TYPES Count % n=28 schools 10 schools 18 schools 
bur vs. prof i 25.2% 34.6% 21.8% 
bur supp prof 26 8.5% 4.9% 9.8% 
prof vs. pol 18 5.9% 8.6% 4.9% 
prof supp pol 19 6.2% 3.7% 7.1% 
market 17 5.6% 0.0% 7.6% 
mor supp prof 16 5.2% 12.3% 2.7% 
bureaucratic 16 5.2% 0.0% 7.1% 
bur vs. mor 15 4.9% 3.7% 5.3% 
professional 15 4.9% 0.0% 6.7% 
political 13 4.2% 0.0% 5.8% 
prof vs. mark 10 3.3% 6.2% 2.2% 
bur vs. pol 9 2.9% 3.7% 2.7% 
bur vs. prof supp pol 8 2.6% 1.2% 3.1% 
bur vs. prof vs. pol 8 2.6% 1.2% 3.1% 
prof supp mark 8 2.6% 1.2% 3.1% 
bur vs. prof supp mor im 2.3% 6.2% 0.9% 
bur vs. prof vs. mor 6 2.0% 74% 0.0% 
bur vs. mark 5 1.6% 0.0% 2.2% 
pol vs. prof supp bur 3) 1.6% 0.0% 2.2% 
bur vs. prof vs. mark + 1.3% 4.9% 0.0% 
moral 4 1.3% 0.0% 1.8% 
Total Incidents 306 100.0% 100.0% 100.0% 


i 


Note. Because these two periods cannot be summed, each of the proportions is calculated independently. 


had to start providing services only to certain kids because we were not making legal testing 
requirements.” The other was a tension between school-developed instruction and district-wide 
curricular mandates. One principal spoke for many: “To maintain a steady vision [means] being 
in every classroom as much as possible, which became more difficult as the system became more 
focused on the system, rather than the needs of [school] people.” 

Three years later similar conflicts remained, albeit only for 39% of the 2007-08 principals. 
“You have to make decisions between how much test prep you’re going to do versus how much 
instruction you do.” Principal Bronte went on to say that by 2007-08 this conflict had become a 
routine dilemma: “[Now] you balance it. You try to do as much of what you consider authentic 
education, and do things that you think are worthwhile, and you [also] do test prep.” Speaking for 
nearly a dozen of his colleagues, Principal Tan described how the tensions between the district’s 
curricular expectations and school-developed interventions had been institutionalized into two 
distinct types of accountability. “The [bureaucratic] accountability is not for my purposes. It’s 
whatever data or statistics they want to show. To me, it’s moving students, graduating them and 
moving them to college. That’s what it’s all about, and making the right decisions for them along 
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the way.” Principal Steinbeck explained that deciding between them was not always an option. 
“T have a boy. ... He’s now in the supposedly tenth grade [15 years old]. No credits this year. 
He comes to school late every day with a basketball . .. sometimes I have to kick him out of the 
buildings at 6:30, 7:00, 7:30, 8:00 o’clock at night. ... He roams the halls. Doesn’t go to class 
_... He does not want to be here or we do not have whatever it takes to motivate him. [But] he 
can’t go anywhere [else]. Most of the GED programs or other types of alternative programs begin 
with children who are 17 years old .. . [and] they have to have X amount of credit.” 


New Forms of Professional and Bureaucratic Conflict 


Principal Eliot and five others added budgets. Cuts—whether routine or unexpected—altering 
even the best plans for improving student performance. “In the past three years [the DOE] hasn’t 
come close to [estimating] an appropriate number [of students] here at Eliot High School and it’s 
affected us drastically twice, where we were short many kids and lost hundreds of thousands of 
dollars in the middle of the year. Just this past month I lost money because we didn’t get enough 
kids and because the mayor cut everyone’s budget. . . . I’m constantly fighting with [and] debating 
the budget office, [saying] ‘I need money for this ...” And sometimes it doesn’t happen. That’s 
when you bang your head against the wall.” 

The conflict between DOE priorities and principals’ professional expectations could become 
especially frustrating when, as Principal Proust put it, “The Department of Education has made 
no choices. It’s just been, ‘Let’s do everything.”’ As a small-school principal she was certain: 
“I can’t do fifteen initiatives at one time because they’re not gonna work, because people are 
gonna be confused. They feel like this administration, I think, does have lot of good ideas, [but] 
unfortunately is trying to do, probably, a lot in a little bit of time, before they are gonna be ousted 
from power. And maybe if they knew that they had a longer timeline, they would be doing it 
more sensibly.” Some felt this tension acutely because their hard-won professional experience 
was discounted if it predated mayoral control of the schools. Principal Morrison explained that 
because “the job has changed immensely” she lacks control over “the real things that impact the 
school.” She asks herself, “Would I do it now, over again, knowing what I know now? I don’t 
think so.” 


Growing Alignment Between Professional and Bureaucratic Expectations 


Reflecting one of the largest shifts we saw—from 20% of incidents in 2004-05 to 72% in 2007— 
08—principals told us critical incidents in which bureaucratic mandates from the DOE supported 
their professional judgments. This change reflects specific support structures: the mandate to 
create a data inquiry team in every school, the SQR process, and unanticipated favors from 
individual district administrators. Together these accounted for 57% of all such incidents in 
2007-08. 

Principal Cather exemplified those who singled out the data inquiry team process. “I love 
the idea that teachers are researchers, because isn’t that what your goal is? Let’s start with 
an open-ended [question] and then let’s see if we can develop some practices that work.” The 
inquiry team is assembled from among the school’s teachers, staff and administrators by the 
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principal and charged with identifying a small subset of students, about 15, facing serious 
problems with performance. It is charged with using the scientific method—examining students’ 
data, hypothesizing interventions to accelerate their learning, conducting the interventions, and 
monitoring the students’ performance—to improve performance. As Principal Cather described 
it, “My inquiry team took 15 students, ninth and tenth graders who scored at a level one or two 
coming from the eighth grade, who seemed to have a problem with fluency. We gave them a 
series of standardized tests, pretests. And then we moved them along and now what we’re doing 
is we’re gonna [look at] January scores in English and global studies.” He, too, is learning how 
to predict student performance: “My first thing was “Let’s see the kids who failed global studies 
in the ninth grade as a predictor.’ [But] there was no correlation . . .so we had to look beyond’.” 

Principal Oates was equally direct about how the SQR process aligned with his professional 
goals. “We find the quality review self-evaluation, which takes place prior to the quality review, to 
be one of the most important things that we do. . .. My three assistant principals and three or four 
other teachers actually sit in a room. ... We are brutally honest with ourselves. ... Every time 
we're self critical, somebody . .. generally me, takes umbrage. . . . ‘If you think there’s something 
we're doing that doesn’t make us the best at what we’re doing, what can we do to make us 
better?”’ In contrast to the preparation process, the quality reviewer’s rating “is either going to 
verify what we said is right, or ...is going to give us a higher grade than what we thought we 
should have gotten, in which case I say ‘Thank you’ and move on.” 

Principal Eliot gave us one typical example of unanticipated support from a district admin- 
istrator that garnered his school an award-winning program. “[The superintendent] spoke to a 
woman who was running the program for him. ... She had been here and visited us .... And 
she suggested us. ... She said he turned to her in the elevator and said “Yes. Let’s put it in that 
school. That’s a good school.’ .. . And just that conversation . . . has dramatically altered the lives 
of some of these kids.” 


Parent’s Altered Influence 


In 200405 conflicts between the principal’s sense of professional accountability and her political 
obligations to the school’s external stakeholders, usually parents, were common. Then, principals 
told stories about parental influence that directly conflicted with their educational goals: For 
example, the pressure exerted by parents to change the schools’ grading policy: “What your 
dealing with is parents who went to school and therefore [think they know] what a school is 
about.” 

Notwithstanding the ubiquity of conflicts between school goals and individual parents’ ex- 
pectations, 3 years later, political accountability was described differently. Several high school 
principals told us stories about using political negotiation and coalition building skills to en- 
gage parents, often by initiating school meetings to discuss problems or establishing routine 
communication via the internet. Principal Hemmingway’s incident was typical. “Last week, I got 
an e-mail from ...the PTA president, which I found very disturbing. [He] basically questioned 
my commitment to the school; and it was based upon misinformation. [Later] when I saw him 
... I took out the data for this school for the period of time before I was the principal, and I took 
out the data for the school during the time for which I was the principal. And I talked about the 
progress we’ve made and I talked about things that we’ve done to bring .. . us that progress. And 
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he was convinced.” Afterwards, Hemmingway reflected, “I think that to be a principal here, right 
now, you have to be . . . highly resilient. You have to be able to take criticism and respond to it 
appropriately.” 

In 2007-08 more principals reported that political accountability supported their professional 
decision making if they expanded their definition of constituents to include community-based 
organizations, businesses and local politicians. “Sometimes when you have a vision and you 
think, ‘I don’t know how I’m gonna do it,” Principal Salinger reflected, “you really gotta reach 
out to people and communicate, because you’d be surprised how willing people are to work with 
you. Sometimes [principals] just don’t do it. It is hard work; it’s a lot of meetings. It’s a lot 
of networking, it’s a lot of begging. . . . These people [who helped with student internships] are 
happy that I called them.” 

Principal Marquez epitomized those principals who had learned how to reach out to politicians. 
When still the assistant principal at her school, she had developed an accelerated program for 
ninth graders that allowed them to graduate early if they attended a longer school day. But the 
program needed a building. She “had to go to every single community meeting” where she would 
“talk up the need for it.” She also testified before the Borough president and in the city council. 
Since then, these politicians recognize her, and several agreed to be guest speakers at events for 
the school. “Politicians all recognize the need. . . . They are all interested in the community.” But 
Principal Angelou found that supportive political patrons could also engender conflict with the 
DOE. She explained, “You know what happens when the aunt that everybody’s afraid of comes 
to town and favors the stepchild? After that aunt leaves, that stepchild gets beat.” 


Competition Becomes a Resource 


In 2004-05, market accountability showed up when in conflict with professional accountability. 
These incidents were relayed as feelings of “pressure” to recruit quality teachers in competition 
with districts outside of the city: “Long Island is literally on my doorstep [offering] higher pay, 
smaller class sizes. They are competing for the very same teachers and university graduates that 
I am, and I’ve lost.” 

When asked if she ever experienced market accountability in 2007-08, Principal Proust’s 
immediate response was, “We aggressively go after teachers. .. . We spend a ton of money and 
effort on teacher recruitment, everything from having fancy materials that make our school look 
better than any other school, and banners. ... My hiring coordinator, you know, emails every 
{education] professor . . . at every major university around the country. .. . We’re in contact with 
the Black Students’ Association.” In her view, the school’s competitive advantages—teamwork, 
a collaborative culture, plus “autonomy in terms of teaching”— did not obviate the need to seek 
out new teachers each year. 

Principals told us more often about harnessing competition to leverage their school’s instruc- 
tional success by seeking out the “right” students. “I’m a competitive person by nature ... so 
I think by nature you are always looking at where you stand in comparison to those in your 
community.” Principal Oates went on to explain how he competed for freshmen. “T study the 
ones who don’t come. .. . [Ours] isn’t always their first choice. There are two other choices they 
[tend to] make. ... So generally we go through all the applications. . . . I get a thousand applicants 
for seventy-five seats. I’m never in a situation where I go begging for kids, I just need the right 
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kids.” Another principal had a mental list of three schools against which he ranked himself each 
time a progress report came out. His list had nothing to do with the peer group established by the 
Department of Education. Parents, he reasoned, looked at their child’s choices differently than 
did the DOE. 

Principal Eliot encouraged competition within the school. “I give out T-shirts. It started with 
the students. We have a positive behavior program. ... And teachers would come to me and say 
‘I want one for this kid who did this.”’ But he insisted that T-shirts only be a reward for results, 
“not for each kid, [rather] the kid who had the highest grade in each class.” Those principals 
who embraced market accountability often said something like, “I find more competition in this, 
driving this, than I do the accountability from the [DOE] ... because I know that if I do this, I 
will never have to sit there and go through accountability checklists.” 


Moral Commitments Take a Back Seat 


Principals in 2004-05 told us that they felt their moral and professional accountabilities were 
aligned four times more often than did principals in 2007-08. How principals enacted moral 
accountability also changed: In 2004-05 it was enacted as a bulwark supporting the principal’s 
professional commitments, whereas 3 years later we heard much less confidence about acting 
on one’s value commitments. For example, Principal Birch revealed that her ethical commitment 
to social justice buttressed her professional commitments to progressive pedagogy, a constant 
motivation over her 20 years as an educator. “I became a teacher to empower those [students 
and families] who maybe didn’t have some access ... we need [access] now more than 20 years 
ago.” Principal Hemmingway described a different relationship in 2007-08. “Sometimes there 
is actually a conflict between the two, moral accountability and bureaucratic accountability.” In 
situations like deciding whether to punish a student infraction by the rules, or bend them to 
meet circumstances, “I guess which way you go depends upon ... what level of risk you feel 
comfortable with.” 

Another form of conflict between moral commitments and bureaucratic requirements, com- 
mon in 2007-08, would not have been possible in 2004-05. Most principals had adjusted their 
expectations to suit bureaucratic, political, and market evaluations of their professionalism by 
2007-08, but they sensed there was something missing. As Principal Morrison described it, “The 
social growth of students is not ... evaluated well enough. I don’t know if there is a mechanism 
for doing it. ... But to me there has to be a way of measuring a kid that comes to the school in 
the ninth grade who may have been left back and is reading on third grade level and is living in a 
shelter, Okay? ... Some of the kids who did not graduate last year I was very close to, and I saw 
them as ninth graders as squirrelly, troubled kids, who, granted, did not graduate in four years, 
but had evolved into a semblance of a functioning adult who could rationalize and reason four 
years later ... and these are kids that are riddled with so many problems of society. .. . And to 
me some of that has to be factored into this scenario somehow.” 

Principal Oates mused about moral compunctions being a vanishing perspective. “I, as most 
people who have worked in the system as long as I have, do become resentful having gone through 
what I think is the only way to go through the system .. . bring[ing] .. . the human point of view 
to the principalship. And I worry that the [moral] accountability . . . will be something that some 
of the people that come after me won’t think is the most important thing.” 
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REFLECTIONS ON PRINCIPAL POWER 


In 2004—05 our principals appeared focused on the school’s internal instructional environment 
and school relationships defined by value commitments. They told us about critical incidents 
that involved conflicts between external types of accountability—political, bureaucratic, and 
market—and their professional and moral commitments. They experienced alignment as their 
professional training reinforced their personal values, and vice versa. Their authority lay in the 
ability to nurture, protect, and buffer a carefully crafted internal school environment from external 
demands (Malen & Cochran, 2008). 

By 2007-08 our principals were externally focused. All five types of accountability became 
distinct, uncontested forces shaping their decisions. Instead of dichotomizing internal and external 
accountabilities, they constructed their professional commitments to support their external ones. 
Their value commitments were decoupled from their professional ones, as they attended less to 
ethical precepts or experienced moral accountability in tension with their other responsibilities. 
Their authority lay in the ability to embrace multiple data sources, using external constituents 
and competitive forces to create a coherent story about the school. 

These changes appear to trump several other sources of variation among high school principals, 
including variation based on indices of the school’s educational task or aspects of its adult 
relationships. If we are correct, our study extends Leithwood and Jantzi’s (2008) observation 
that district policies explain a large proportion of the variation in how principals see their own 
effectiveness by changing what principals attend to in the accountability environment and how 
they relate one obligation to another. 

How can this be so if, as research suggests, officials “cannot mandate what matters” 
(McLaughlin, as cited in Louis, 2007, p. 108); if school leaders make deliberate decisions about 
how to react to the incentives and sanctions they face (Honig & Hatch, 2004); and leaders are 
constrained by their own historically contingent narratives about “our school” (Lemons, Lischei, 
& Siskin, 2003)? The fact that none of our principal informants in 2007-08, although randomly 
chosen, had more than 6 years of experience as a principal hints at one reason for the swift change 
our data depict. It can be a rival hypothesis challenging our conclusion, or as we argue below, a 
contributing factor to a changing politics of the principalship. 

Large numbers of new principals have been recently hired in New York City and anecdotal 
evidence suggests that many seasoned principals have left, perhaps in reaction to the new conse- 
quences assigned to bureaucratic accountability. Many of the newcomers lead small schools with 
external partners, share space with other schools in the same building, and are assigned ambitious 
outcome targets for student populations small enough to encourage wide year-to-year perfor- 
mance fluctuations. The professional commitments of these new principals are being shaped in a 
policy environment that actively rewards attention paid to external accountabilities over internal 
ones. External accountabilities become Strategic resources. Embracing district-mandated tools 
(e.g., inquiry teams, SQR) helps principals describe how their professional priorities align with 
district goals. Exploiting an advantage in the competition for scarce human resources replaces 
moral compunction as a motivation to excel. Mastering political negotiation skills helps ensure the 
perception of progress, even if some targets are not met. If this narrative is reasonably accurate, it 
suggests that simultaneous policy changes appear to have altered the politics of the principalship. 

Most analysis has framed the politics of the principalship as a series of efforts to contain 
or manage conflict in the face of mandates from policymakers and occasional demands from 
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middle-class parents and activist teachers.'* Our findings suggest that this focus on the differential 
forms of power that principals wield to control the internal school agenda are too limited. They 
may miss the ways that external interests and constituencies are now shaping principal decisions, 
including how those external pressures realign principals’ professional expectations. 

Those studying the politics of the principalship have routinely argued that it requires special 
micropolitical lenses to capture. But if the behaviors we have identified persist and prove gener- 
alizeable, we may make fewer analytical distinctions. Perhaps principals should be studied like 
other appointed political leaders, complete with constituencies and special interests to satisfy; 
self-interested, competitive motivations; and professional standards of practice that may align 
with their superior’s expectations as well as conflict with them. We might ask, What are the rela- 
tionships between the private interests influencing policy changes and the public authority vested 
in the principal? How do principals coopt or cooperate with civic and political leaders? How 
does competition for scarce human resources in a high-stakes environment reshape principals’ 
strategic decisions? What is lost if principals no longer expect their professional obligations to 
spring from their moral commitments? 

Mayor Bloomberg recently succeeded in obtaining a one-time exemption from the city’s term 
limit law and has as much as $100 million to spend on reelection for a third term in 2010. The 
city’s mayoral control law looks set to be reauthorized with relatively minor changes in 2009. 
Past may well prove prologue for at least 4 more years, and the politics of the principalship may 
be shifting before our eyes. 
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Teachers unions are among the most powerful, yet least studied, actors in public education today. 
Although public attention focuses on the influence of national unions, the policies that most affect 
teachers and schooling are bargained by local unions and school boards. Interviews with 30 recently 
elected local union presidents reveal that these leaders balance competing interests and obligations, 
from the concerns and priorities of their members to pressures from state and national affiliates. 
Although these presidents reaffirm the traditional union agenda, most also advocate an expanded 
agenda of teacher professionalism and more collaborative approaches to collective bargaining and 
contract management. 


Teachers unions are widely recognized as powerful forces in public education. Union critic 
Terry Moe (2001) asserted, “The teachers unions probably have more influence on the public 
schools than any other group in American society” (p. 151). Despite their acknowledged im- 
portance, scant research exists to explain how unions exert their influence and what effects they 
actually have (Hannaway & Rotherham, 2006). The media and union critics commonly portray 
teachers unions as monolithic organizations that pursue traditional union goals and a narrow 
agenda of self-interest. Moe (2006) shared this view, arguing that “if public education is to escape 
the stultifying drag, of the unions’ grip on the system—and if the system, therefore, is to evolve 
into a new form that is better suited to providing a quality education to children—it will happen 
only through reforms that weaken or eliminate union power in the schools” (p. 230). 

Despite such widely held views, national leaders of the American Federation of Teachers (AFT) 
and National Education Association (NEA) cannot effectively dictate the priorities or practices of 
local union presidents. Forty-five states permit teachers to organize locally and bargain directly 
with their school board. Every contract is negotiated separately and must be ratified by local 
teachers. This process, often criticized for its inefficiency and redundancy, is consistent with a 
long tradition of local control in U.S. public education. 
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Every local union elects a president who is well positioned to influence teachers and schooling 
by shaping local priorities, building coalitions to achieve shared purposes, negotiating the contract, 
and working either collaboratively or contentiously with school officials. Local union presidents 
must balance a number of competing interests and obligations. To be reelected, they must convince 
a majority of their members that they successfully represent those members’ interests and views. 
At the same time, most local unions are relatively small organizations, making it necessary 
for their presidents to rely on state and national affiliates for political and legal resources. 
Meanwhile, these presidents have their own views and priorities, which may or may not align 
with those of their members or affiliates. It is the approaches and actions of these presidents, who 
operate in an increasingly challenging context for both public education and public sector unions, 
which cumulatively determine the impact of unions on schools and efforts to improve them. 
However, policymakers rarely recognize these local presidents’ independence or acknowledge 
their importance, whereas researchers largely ignore them. 

We conducted this interview study to better understand local union presidents and their work. 
What are their views and priorities? How do they exercise leadership, particularly in the dynamic 
context of public education today? What relationships do they build with school administrators, 
who are often cast as their adversaries in collective bargaining, but may be crucial allies in the 
context of school improvement? How do the presidents reconcile or manage differences between 
the positions of the local organization and its affiliates? This study was designed to explore such 
questions. 

In the following discussion, we review research about local unions and describe our study. We 
then present and discuss our findings. Overall, we find that the local union presidents in our sample 
defined their own priorities and set their own course of action rather than following the lead or 
dictates of their state or national affiliate. Often, they pursued an agenda that was not limited 
to traditional topics of bargaining, but rather extended more broadly to include issues such as 
teachers’ roles, evaluation, professional development, or school governance. Rather than engaging 
in hostile combat with management, many of these presidents used collaborative approaches to 
both negotiations and contract management. In this way, the presidents in our sample often 
championed and, in some cases, created important educational reforms in their districts. As they 
respond to the changing realities of public schooling and extend their sights beyond traditional 
union interests, local teachers unions represent an important source of educational leadership in 
this period of reform. 


PRIOR RESEARCH ABOUT LOCAL TEACHERS UNIONS 


The few studies conducted since 1970 suggest that local unions have not become the uniform 
organizations that many expected at the outset of collective bargaining in the late 1960s. Instead, 
teachers’ contracts have become more varied. Local unions do not respond in lockstep compliance 
with the dictates of state and national affiliates, confine their bargaining to traditional issues, or 
rely exclusively on adversarial labor relations. 


The Influence of National and State Unions on Local Affiliates 


State collective bargaining laws adopted in the 1960s and 1970s define the process by which 
teachers organize, choose a union to represent them, bargain contracts, and settle disputes. 
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Modeled on the 1935 National Labor Relations Act, these laws led to contracts and practices that 
were similar across states and districts (Johnson, 2004). However, this similarity stopped well 
short of uniformity. 

During the first 2 decades of collective bargaining, researchers who analyzed teachers’ con- 
tracts found that they generally covered the same topics. McDonnell and Pascal (1979) analyzed 
151 contracts and reported a “convergence of collective bargaining outcomes over time. As more 
and more school systems follow the lead of flagship districts, there is less variation among in- 
dividual contracts” (p. 31). Five years later, Goldschmidt and Stuart (1984) analyzed another 
sample of contracts and reached a similar conclusion, leading some to observe that national 
unions were gaining control of local school governance. However, when McDonnell and Pascal 
(1988) returned to their original sample of 151 contracts nearly a decade later, they found far 
more variation than they had expected, a finding subsequently confirmed by other studies (Ballou, 
2000; Johnson, Nelson, & Potter, 1985; Hess & Kelly, 2006). 

Findings about local unions’ compliance with their state and national affiliates tell a similar 
story. Throughout the first 2 decades of collective bargaining, locals looked to these affiliates for 
guidance about what and how to negotiate and many went on strike when they reached impasse 
with their school board (Johnson, 1987). Often such strikes were orchestrated by state union 
representatives. However, in the 1980s, conflicting views began to surface about what policies 
the unions should promote and what actions they should take. National leaders, such as Albert 
Shanker of the AFT and Bob Chase of the NEA, advocated progressive policies, which some state 
and local leaders vigorously opposed (Fuller, Mitchell, & Hartmann, 2000). Meanwhile a group of 
local presidents from both the AFT and NEA allied to “promote new union models,” forming the 
Teachers Union Reform Network, whose members sought alternatives to narrow and conventional 
union practices (Urbanski & Erskine, 2000). Thus, although no studies systematically examine 
the extent of conformity across union organizations, case studies reveal the ongoing debate within 
teachers unions about what kind of organizations they should be and whether they should endorse 
traditional or progressive positions (Kerchner & Koppich, 1993). 


The Breadth of the Labor Agenda 


Each state’s collective bargaining law specifies the scope of issues that a district must or may 
bargain. Some define a narrow scope of bargaining, which obliges school boards to negotiate 
only about wages, hours, and basic working conditions. Others define a broad scope, requiring 
the parties to bargain about a wide range of specific issues, such as evaluation and in-service 
training. Some statutes also permit districts to consider other matters beyond the mandated scope, 
if both sides agree to do so. Overall, the required scope of bargaining focuses on matters of 
teacher welfare—for example, pay, layoff procedures, or the length of the school day—whereas 
broader professional concerns about curriculum or school governance remain permissive top- 
ics for negotiation. Early studies of teachers’ contracts found many provisions about topics 
that the state had not authorized districts to bargain, suggesting that the legal scope of bar- 
gaining may influence, but does not limit, what local districts negotiate (Goldschmidt, Bowers, 
Riley, & Stuart, 1984; McDonnell & Pascal, 1979). Kerchner (1978) concluded that over time 
the scope of bargaining expands both because of legal interpretations and because the local 
parties choose to bargain about a wider range of topics. Although they operate in a strong 
culture of labor relations, which reinforces traditional beliefs and practices about collective 
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bargaining, local parties have more influence than state governments in determining what the 
contract covers. 

Researchers, such as Kerchner (1986), also have examined whether local districts hold fast 
to a traditional agenda or pursue a more professional or reform agenda. Johnson and Kardos 
(2000) argued that before the mid-1980s, most local districts negotiated contracts that were ad- 
equate for running factory-model schools where “teachers serve as workers in a production-line 
process, delivering standardized education to students” (p. 10). However, in the 1980s some 
local unions recognized the need for more flexibility in schools and the rules that regulate them. 
They introduced contract provisions that accommodated reform, for example, by establishing 
school-site councils, instituting peer review, authorizing joint labor-management committees to 
allocate staff to schools, creating career ladders for teachers, or limiting the role of senior- 
ity in teacher assignment (Kerchner & Koppich, 1993; Rosow & Zager, 1989). Since 1986, 
many local districts have experimented cautiously with “reform unionism,” others have em- 
braced it, and some have rejected it altogether (Johnson & Kardos, 2000; Kerchner & Koppich, 
2000). 


The Character of Local Labor-Management Relationships 


The process of collective bargaining is explicitly bilateral and often adversarial. Parties imple- 
menting collective bargaining laws in the public sector often assume that labor and management 
interests are at odds and that the parties must compete for a limited pool of resources—what one 
side wins the other loses. State laws specify the details of this relationship, citing in formal terms 
the rights and obligations of labor and management. Given the history of industrial negotiations 
and the legal structures that frame it, many expect local labor relations in education to be highly 
routinized and combative. Therefore, it was particularly noteworthy in the mid-1980s when many 
districts began to negotiate using collaborative approaches. 

Perry and Wildman first identified such approaches in their 1970 study of 24 local districts. 
Fisher and Ury’s (1981) Getting to Yes prompted many local labor and management teams to 
pursue “interest-based” or “win—win” bargaining, a process encouraging the parties to explore 
their common interests and develop solutions to shared problems. Such processes were used 
widely to adopt reforms such as peer review, site-based management, and new professional roles 
for teachers. Two sets of case studies (Kerchner & Koppich, 1993; Rosow & Zager, 1989) recount 
the development of some of these collaborative efforts in which the labor-management lines were 
blurred and joint committees created and managed programs together. Kerchner and Koppich 
(1993) reported that such committees were “universal” in reforming districts, such as Rochester, 
Miami-Dade, and Cincinnati. 

Thus, empirical evidence suggests that the practices of collective bargaining have evolved since 
the 1960s to include more varied and collaborative practices. Yet Julia Koppich (2006), who has 
documented much of this evolution, asserted that teachers unions have yet to be fully transformed. 
She suggested that the increased demands of standards-based accountability, competition from 
charter schools and vouchers, and large-scale turnover in the teaching force are changing and 
challenging the work of local union presidents. We conducted this study in an effort to better 
understand these presidents’ views and their work in this new context. 


378 S. M. JOHNSON ET AL. 


THE STUDY 


We sought to understand the thoughts and approaches of the newest generation of local union 
leaders—those who were elected during the past 8 years. Specifically, we were interested in these 
leaders’ priorities and bargaining agendas, how they would describe local labor-management 
relations and how they would characterize their relationships with their state and national affiliates. 

We first selected a sample of six states (California, Colorado, Florida, Massachusetts, Mary- 
land, and Ohio), which varied from one another in important ways—geographical location, state 
labor laws, and political context. Within each state, we chose a diverse sample of five districts, 
varying in size, type (urban/suburban/rural), student enrollment trends, and national union af- 
filiation (NEA/AFT). We explicitly sought to study a group of union leaders whose views and 
activities ranged from traditional to reformist. 

Between March and September 2006, we conducted 2-hr, face-to-face interviews with 30 
presidents in 30 districts (5 per state). By talking with presidents in a variety of settings, we could 
gain a broad perspective on the views of local union leaders today. However, because we did not 
interview other actors in each district, we could not verify or elaborate on the story presented 
by any single president. Therefore, we supplemented the interviews with systematic analysis of 
the local contracts and newspaper accounts, which provided further evidence of local policy and 
practice. Ultimately, though, this is a study about the presidents and their views. Because this 
sample is relatively small and not randomly chosen, the findings, though instructive, cannot be 
generalized. It is, however, the first major study of local union presidents and therefore provides 
4 foundation for further research about this critically important, but largely unstudied, area of 
education policy.’ 


The Presidents 


Before we began this work, we thought that many recently elected union presidents might be 
early-career teachers, and we expected to find distinct differences between “old-school” and 
“new-school” leaders. However, no such patterns emerged. As a group, these presidents were 
seasoned teachers, much closer to retirement than to entry. They had taught between 7 and 37 
years, had an average 25 years of experience, and ranged in age from 29 to more than 60. This 
group was nearly balanced by gender (14 men and 16 women) and included 22 Whites, 5 African 
Americans, and 3 Hispanics. 

The presidents were not only experienced teachers, but virtually all were also longtime union 
members, who had developed as union leaders over many years. As loyal insiders, most first 
became active in the union as building representatives and then worked their way up through the 
ranks. Most said that they ran for president because they were next in line for the job, having 
served in a key role, such as bargaining chair or vice president, during the prior term. However, 





‘An earlier report based on this study (Johnson, Donaldson, Munger, Papay, & Qazilbash, 2007) documents the 
presidents’ views of and approaches to leading two generations of teachers; engaging in collective bargaining and 
contract management; reforming teachers’ pay; and advancing teacher quality. This article focuses on their priorities 
and practices as leaders, examining how they respond to their affiliates’ priorities and members’ concerns as they set a 
direction for their organization. 
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some of our respondents had defeated an incumbent president or mounted a successful challenge 
to the heir apparent. These presidents explained that they had decided to run out of turn because 
they disliked or distrusted their opponent, disagreed with their union’s current stance, or wanted 
to rescue the union from its inside circle of leaders. A few presidents, all from smaller districts, 
had simply agreed to take the job when no one else would. 


MANAGING THE RELATIONSHIP WITH STATE AND NATIONAL 
AFFILIATES 


Contrary to widely held beliefs that national and state unions control their local affiliates, the 
presidents we interviewed said they remained largely in charge of their own affairs. The state 
and national unions provided them assistance and advice—but not prescription—in political 
action, legal services, professional development, and priority setting. For the most part, lo- 
cals felt able to adopt, modify, or reject policies and positions crafted by the state or national 
unions, and they did. Moreover, most presidents said that influence was mutual. The locals 
could play a role in defining state and national positions—especially in the political arena—by 
choosing whether or not to participate in initiatives organized by the affiliates. Thus, presidents 
rarely reported feeling bound by the expectations of state and national union officers. Presi- 
dent Jim Groth in Chula Vista, California, said, “I know the ins and outs of CTA [California 
Teachers Association]. And I know when to tap them and when to tell them to just leave me 
alone.” 


Lobbying and Political Action 


In carrying out lobbying and political action, locals and their affiliates participated in a give- 
and-take process that was mutually beneficial. Presidents said they expected lobbying support 
and services from the state and national unions in exchange for the substantial membership dues 
that they contributed to these organizations. The presidents recognized that the state union staff 
had greater capacity and was better positioned to lobby state legislators about issues such as 
education funding, allocations for teachers’ salaries, or changes in the state education code. In 
turn, the locals provided the manpower to conduct public protests and door-to-door campaigns. 
Presidents in Maryland described how their state union coordinated a successful campaign for 
pension reform, which established a defined benefit retroactive to 1998. Frederick County’s 
Gary Brennan said, “The state association kind of led and organized it, but a lot of it had to 
be organized by the locals. The locals worked together.’ Similarly, California presidents told 
of participating in their state union’s organized opposition to three propositions championed by 
Governor Schwarzenegger, which would have lengthened the time to tenure, restricted the use of 
union dues for political activities, and increased the governor’s power to reduce school funding. 
Bruce Seaman in Grossmont, California, recalled walking “precinct by precinct to talk with 
people, to get them out to vote against [the propositions].” 

Heads of some locals said that they sought support from affiliates when they encountered 
political challenges. Irma Valerio of Colorado Springs told of calling upon NEA president Reg 
Weaver to visit her district three times when Colorado Springs was at the “center of ... [a] 
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national bull’s eye” in a dispute about vouchers: “They acknowledged that we needed some other 
support.” A. J. Duffy, president in Los Angeles, asserted that the local and state organizations are 
“absolutely separate. But the bonds that have been created are very strong and I intend to keep 
them strong.” He told of insisting that the president of the California Teachers Association, “the 
largest state affiliate in the country,” sit beside him during “discussions with the mayor, face to 
face, about mayoral takeover. It’s nice to have that kind of muscle that you can call upon ... ie 

In a few cases, local leaders objected to their state affiliate’s stance but nevertheless fell in 
line because they had to count on the larger organization’s support. Ann DeLacy of Howard 
County reported that the Maryland State Teachers Association (MSTA) made what she called 
“unreasonable” demands for lobbying and endorsing certain state-level candidates. When her 
local wanted to endorse all candidates running for state office, the MSTA said they could not. “It 
was a major fight. And so because of funding, we decided to go along with the MSTA, but then we 
would let people know that we strongly supported the other candidate as well.” DeLacy reported 
that, by contrast, the nearby Montgomery County union could ignore the MSTA’s demands 
because it had “lots of money.” 

Thus, local teachers unions sometimes requested assistance from their state and national 
affiliates on political matters. In return, they provided what the people needed to ensure that 
the affiliate’s political initiatives would succeed. Most presidents said they were free to request 
advice and support when they found it necessary, though financial realities enabled some local 
unions to be more independent than others. 


Legal and Bargaining Assistance 


Presidents said they relied on their state organizations for legal advice and bargaining assistance, 
in part because many presidents and their leadership teams taught at least part-time and lacked 
the time and expertise to do this work on their own. However, even large unions with full-time 
presidents asked their state affiliate for expertise in specific areas. 

Virtually all presidents said they turned to their state affiliate for legal advice. In Colorado 
Springs, Valerio regularly asked her state union consultant about legal matters: “I have him on 
speed dial’? Needham, Massachusetts, president, Sherrill Neilsen, herself a former corporate 
lawyer, had full-time teaching responsibilities. She regularly consulted her state union’s lawyers 
who understood “all the ins and outs about ed[ucation] law in Massachusetts.” She noted, “So 
any time there’s an education law question, I just contact the UniServ rep and within a couple 
of hours, she will have printed out all the relevant provisions of Massachusetts General Law. I 
could go and spend my time, but I don’t have time. And I’m not up to date on all that, and that’s 
not my background.” Some unions even had a state union consultant working on site with them. 
For example, Tom Lynch, of Westminster, Colorado, and Mark Chavez, of Boulder, Colorado, 
shared an onsite UniServ representative, who could provide assistance in a range of matters. 

Presidents—especially those in smaller districts—also relied on their state affiliate for in- 
formation and assistance in bargaining. Seaman in Grossmont, California, said that the state 
organization provided “two very top-notch negotiation specialists” who worked with him during 
fact-finding, if bargaining reached impasse. Glades, Florida president, Janice Brown, said that she 
called upon the state union primarily for bargaining assistance. However, like other presidents, 
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Brown said that she also regularly contacted presidents of neighboring unions for information 
and advice. The formal and informal networks supported locals in bargaining. 


Professional Development 


Many presidents said that they also valued the training and conferences sponsored by their affili- 
ates. Several Florida presidents spoke about the AFT’s Educational Research and Dissemination 
(ER&D) “train the trainer” professional development program. Collier County’s Von Jeffers de- 
scribed his union’s extensive offerings of workshops and modules based on ER&D trainings, 
including, for example, an introduction to research-based foundations of reading instruction. 
“The actual professional development that I’ve received from the district has been very poor, in 
my opinion. But the professional development from [ER&D] has been fantastic. It’s a really good 
program.” 

Several presidents in Massachusetts praised the state union’s summer programs for new 
teachers and union presidents. Timothy Sheehan from Amherst-Pelham, the youngest president 
in the study, said he had become active in planning the state’s conference for new teachers, a step 
that led him to run for president. 


Influence Over Local Bargaining Priorities 


Although presidents often relied on their state and national affiliates for lobbying and political 
action, legal advice, bargaining expertise, and professional development resources, they were 
less likely to turn to those organizations for guidance about their bargaining priorities. Instead, 
they crafted their agenda largely in response to needs they identified in their union, schools 
and communities. Many were aware of the state or national affiliate’s stance on issues such as 
performance-based pay or class size, and a number said they were influenced by these positions. 
However, none said that their state or national organization dictated their local bargaining prior- 
ities. Although the local leaders ultimately made independent decisions about what to bargain, 
some did feel pressure to conform with their affiliates’ positions. 

If a local union adopted a controversial program opposed by its affiliate—such as using value- 
added assessments in compensation—the president might well encounter resistance. In such a 
case, Paul Toner of Cambridge, Massachusetts, acknowledged, “You wouldn’t be welcome at 
some statewide events, or you would be made to feel uncomfortable.” Asked if he would be 
constrained in any way from advancing such a proposal, Toner said, “Formally, no. It’s not like 
we’d get thrown out of the union or anything like that. It’s a matter of, do you want to be the first 
one? Do you want to go against what appears to be the general premise of the state affiliation?” 
Toner, though, also noted that the NEA did not mandate local compliance with its platform: “I’m 
glad the NEA stays away from taking positions that say: “Thou shalt not do this or thou shalt not 
do that,’ because then it leaves everybody the freedom to be innovators and try things out.” He 
recognized the need for locals to set their own agendas. Similarly, Theo Harris in Palm Beach, 
Florida, remained in contact with state and national union leaders, but did not feel bound by them: 
“Some of it’s just touching base and making sure that you’re on the right track, you’re not way off in 
left field. But I don’t mind being way off in left field sometimes, because that’s part of leadership.” 
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Many presidents, notably those in larger districts, had gone their own way when the state 
or national union’s position did not suit them. These presidents generally worked full-time on 
union business and had a relatively large staff with considerable financial resources. For example, 
president Rhonda Johnson from Columbus, Ohio, said that her state affiliate is not actively 
involved in reform: “We don’t always agree” with the Ohio Education Association and the NEA 
because they are “opposed to things that we’re willing to do.” Montgomery County’s president, 
Bonnie Cullison, said that dues from her local, the largest in the state, accounted for one tenth 
of the state union’s budget, making the affiliate financially dependent on the local: “We are left 
to do what we want to do.” She explained that her union is involved in a “progressive agenda 
focused on teaching and learning. It’s something that makes my state affiliate very uncomfortable. 
... [None] of the leaders at the state level ... are talking about the same kinds of agendas that 
we are.” By contrast, president Sue Taylor from Cincinnati, Ohio, explained that her local union 
supports many of the Ohio Federation of Teachers’ initiatives. However, she noted, “It really 
depends on what the issues are. . . _There’s always going to be some distance and some lack of 
100 percent trust.” 


PURSUING AN EXPANDED AGENDA 


As local presidents crafted their agenda to respond to their members’ needs and advance their 
organizations, most took a twofold approach: They both reaffirmed the traditional agenda and 
moved beyond it. The traditional agenda of teachers unions mirrors that of industrial unions— 
securing better salaries, benefits, and working conditions for members and protecting them 
from unfair treatment. Although the presidents we interviewed said it was essential to pursue 
these goals, few stopped there. Most said that conventional union priorities were necessary, but 
not sufficient, given changes in the context of education: the increasing expectations of new 
teachers for professional support, the demands of accountability and school reform, and growing 
competition from charter schools and vouchers. Therefore, many presidents sought to move 
beyond this baseline agenda, promoting teachers’ active role in reform. Priorities varied, but these 
presidents’ expanded agendas often included initiatives to improve instruction through teacher 
evaluation and professional development, active engagement in school reform, and innovations 
in teacher compensation. These presidents found it a challenge to achieve the right balance 
between the traditional and reform elements of their agenda. As leaders, they could not simply 
advance their own ideas about what the union should become. Rather, they had to consider what 
was possible within their local context and what positions their members might reasonably be 
expected to endorse. 


AFFIRMING THE BASELINE AGENDA 


As the presidents reflected on the purposes of local teachers unions, each mentioned traditional 
labor concerns. For example, Brown of Glades County said the union’s “main purpose is to 
protect teacher rights and get us as many benefits, including good salaries, as possible.” In Los 
Angeles, Duffy said that the union’s first purpose was to “protect the rights and standard of living 
of its members. That’s a time-honored tradition.” Similarly, Palm Beach County’s Harris said, 
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“I believe that you should protect your members [and] make sure that the contract is honored 
and it is properly interpreted and properly followed.” The union, he explained, should “represent 
[members] as it relates to the kind of benefits and wages and working conditions that exist for 
them, to try and make those as positive for your members as possible.” 

Presidents in districts experiencing large budget cuts and teacher layoffs said that job security 
was especially important. In Columbus, for example, where 400 teachers received layoff notices in 
2004, Johnson explained, “So right now, people are in survival mode. If we don’t take care of their 
welfare and they don’t have a job, they could care less about professional development or anything 
else on the academic program.” The context required a focus on bread-and-butter issues first. 

In describing their union’s agenda, though, only a few presidents confined their discussion to 
traditional purposes. One such president, Susan Brooks from Mount Healthy, Ohio, said that her 
views were shaped by her experience writing the original contract and serving as grievance chair: 
“My perspective is [that] the union is here to protect the members of our union, make sure that 
the contract is followed, and to enforce the contract.” Willie Terrell in Dayton said simply, “The 
purpose is to advocate for teachers.” 


Moving Beyond the Baseline Agenda 


Although traditional union purposes served as the foundation of their agenda, nearly all of the 
presidents in our sample said that standard union goals were not sufficient, given current demands 
on public education. Frederick County’s Brennan said, “I think I still have a somewhat traditional 
view. I still think what our members want from us has to do mostly with pay, benefits, workload, 
kind of traditional sort of things. But I also feel we have a responsibility to be champions of public 
education in general.” In Baltimore, Marietta English said, “Today [your vision] has to be more 
than just working conditions, benefits, and salary. You have got to have more than that.” Denver’s 
Kim Ursetta agreed, explaining that in addition to “protecting teachers’ rights,” the union had to 
be “the source for professional knowledge, information, expertise, and growing your own, like 
the old guild model.” 

Because some teachers—especially veterans—took issue with an expanded bargaining agenda, 
presidents often had to educate their members and exercise a new kind of leadership. For example, 
in the context of a threat by charter schools, Cincinnati’s Taylor starkly framed the challenge of 
improving students’ performance: “What I say constantly in our membership meetings is that 
we, first and foremost, have got to find a way to raise student achievement .... And if we don’t 
figure out how to make improvements in student achievement, we’re not going to have a school 
district, much less a union to advocate for anyone.” 

Most presidents in this study expanded the traditional union agenda, talking about the need to 
strengthen quality standards in professional evaluations, to improve instruction through induction 
programs and professional development, and to revise the single salary scale with new strategies 
for teacher compensation. In the process, they sought to elevate the professional side of teaching. 


Raising Quality Standards Through Professional Evaluation 


One of the most persistent criticisms of unions is that they block dismissal of weak teachers using 
excessive procedures and a hard-hitting legal defense of any teacher the district tries to fire. State 
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collective bargaining laws oblige unions to fairly represent any teacher in the bargaining unit 
(union member or not) whose contract rights are violated. However, the union is not required to 
support a teacher who disagrees with the substance of her principal’s evaluation or the district’s 
decision to dismiss her. A number of presidents explained that they and their executive committees 
have considerable latitude to decide whether to defend a teacher if the evaluator has followed 
the negotiated procedures. Among the presidents we interviewed, very few reported that their 
union automatically defends any teacher whom the district moves to dismiss. Voicing a common 
sentiment, Melissa Cropper of Georgetown, Ohio, said, “I don’t think the union should be 
protecting people who don’t belong in the profession. . . . I protect procedures. If an administrator 
wants to get rid of a teacher, then there is a procedure to follow for that. And if that procedure is 
not followed, then the union has to step in and protect that. But I don’t fight for a bad teacher to 
be kept in the district.” Similarly, Collier County’s Von Jeffers said, “We still make sure that they 
have due process. But [the idea that] we only exist to protect bad teachers—we’re working real 
hard on getting rid of that myth.” 

Toner, a lawyer and union president in Cambridge, Massachusetts, explained the limits of the 
union’s obligation in a memorandum to his members. He told us, “All we can do is hold the 
school department accountable for due process. And guess what? They are following the process, 
a process that was negotiated in the contract and [that the teachers] are well aware of.” Toner 
emphasized, however, that district officials also have obligations; they cannot “ignore somebody 
in a classroom for two years and then say, “Oh, you are not meeting expectations.” 

Several districts within our sample had completely changed the traditional relationship between 
labor and management in performance evaluation. The Peer Assistance and Review (PAR) pro- 
grams in Cincinnati, Columbus, and Montgomery County represented the most intensive efforts 
to ensure teacher quality by reforming supervision and evaluation. PAR in these three districts 
combined peer mentoring and assessment into one program. Consulting teachers, assigned to as- 
sist both new and experienced peers, eventually recommended whether those teachers should be 
reemployed. In virtually all cases, the joint labor-management panel that ran the program upheld 
their recommendations. Montgomery County’s president, Cullison, emphasized the program’s 
importance in supporting and assessing new teachers: 


If they’re going to come and teach in my district for five years, I want to make sure they’re really 
good. I want to make sure that, if I have to go to the mat for them, I can say this is someone who has 
given her heart and soul to the kids in Montgomery County... . But I’m not going to do that for just 
anybody. I’ve got to have some level of assurance that these are highly skilled people for me to be 
advocating for, which is why PAR is really, really important to me. 


Improving Instruction through Induction and Professional Development and 
Reform 


Unions in most of the 30 districts also were actively involved in efforts to enhance teacher perfor- 
mance. Although professional development is widely seen as a management responsibility, more 
than half of the union presidents we interviewed were developing or promoting union-sponsored 
induction programs, professional development programs, or both. On average, these presidents 
listed “professional development” as their third most important priority, behind salary and benefits. 
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In a few districts, local unions worked jointly with district officials to provide mentoring for 
new teachers. Some also provided short professional development courses for teachers, as both 
a means for instructional improvement and a leadership strategy to attract new teachers to the 
union. Carol Kilby in Prince George’s County said she hears “from the young teachers. They 
hunger for professional development .... I’m being won over to that. I really see how much it 
has of value and how it does link them up to the union.” To varying degrees, all of the Florida 
unions in the sample sponsored professional development. Jeffers, from Collier County, Florida, 
used these courses in his pitch to new teachers: “If you want to improve your teaching, then join 
the union because we can offer that service to you.” 

Most of the unions that sponsored their own professional development were large organizations 
with considerable resources. However, Cropper in Georgetown, Ohio, one of the smallest unions 
in our study, also decided to lead on this issue, explaining, “I very much see the union as being a 
proactive organization.” With the superintendent’s endorsement and broad teacher input, Cropper 
planned the following year’s 4 days of professional development to help teachers use technology in 
instruction. Although this president’s initiative was unusual, virtually all of those we interviewed 
expressed interest in having the union involved in their members’ ongoing development as 
teachers. 

Many presidents wanted their teachers to have a meaningful voice in instructional reform. 
Pomona’s Sarah Ross laid out an expansive professional agenda: “I believe the union’s here to 
ensure that teachers have a voice in what’s happening in their classrooms, in their schools, and in 
the curriculum, so that we can best meet the needs of our students.” Boulder’s Chavez echoed her 
concerns: “We need to make certain that the teachers do have a strong voice in the profession, 
because if we don’t, someone else is going to tell us what we should do.” 


Revising Teacher Compensation 


Presidents often said that they did not fundamentally oppose performance-based or alternative pay 
plans, even though state unions had not endorsed—and sometimes opposed—them. In fact, many 
of their districts were experimenting with modifications to the standardized salary scale. Some 
had introduced relatively modest changes that applied to all teachers; others were experimenting 
with far more substantial reforms in a small number of low-performing schools. In Montgomery 
County, teachers in specialized roles, such as staff developer or consulting teacher in the PAR 
program, could earn extra stipends of $5,000 to $10,000. Teachers in Columbus could earn an 
additional $2,500 when their students achieved performance goals. Cincinnati offered school- 
based performance incentives and rewards. Miami-Dade offered extra pay for a longer school 
day and year in high-need “Zone” schools. 

In Denver, the union had partnered with the district to develop the most far-reaching alter- 
native compensation plan, which eliminated the standard salary scale and included elements 
of performance-based pay. President Ursetta said, “We jokingly call ourselves the black sheep 
of CEA [Colorado Education Association].” Despite being a member of the NEA Resolutions 
Committee, Ursetta endorsed pay reforms that “are not philosophically in line with the beliefs 
of the NEA.” As a result, the state organization “prohibited ...the state staff from helping us 
with any portion of this,’ despite the fact that Denver sends half its members’ dues to the CEA. 
Of the districts we studied, only Denver had actually replaced its standardized salary scale with 
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an alternative compensation system. Other presidents remained unconvinced that Denver’s plan 
was a comprehensive and effective alternative to the standardized salary scale, although others 
expressed interest in that possibility. 


TRANSFORMING THE LABOR-MANAGEMENT RELATIONSHIP 


Although union presidents can promote progressive agendas, they cannot implement new pro- 
grams and practices single-handedly. Instead, district policies are established first through col- 
lective bargaining and subsequently through ongoing contract management. Today’s education 
environment provides strong incentives for the union and management to work together on behalf 
of better schools. Ultimately, both the teachers unions and the schools face the prospect of not 
meeting required targets on accountability tests, losing a large share of the education market 
to competitors, or failing to attract and retain a staff of highly qualified teachers. In fact, the 
presidents in many districts described evolving, collaborative approaches to both collective bar- 
gaining and contract management—approaches that differ markedly from those that prevailed 30 
years ago. Although each of the presidents described a Jabor-management relationship with ups 
and downs over time, their accounts suggested an overall decrease in hostility and increase in 
cooperation. 


A Hybrid Approach to Collective Bargaining 


More than three fourths of the presidents reported that their districts had moved away from 
traditional, adversarial bargaining, which pits labor against management in a zero-sum game. 
This finding is consistent with data showing a steady decline in the number of teacher strikes 
nationwide over the past 3 decades, from a high of 241 in 1975 to only 15 in 2003 (Hess & Kelly, 
2006). Apparently, in today’s context, the costs of resorting to hostile, adversarial tactics often 
exceed the possible gains. 

Several presidents described a period when their districts tried to conduct all their bargaining 
with interest-based approaches, such as those promoted by Fisher and Ury (1981), although only 
two (Palm Beach County and Boulder) said that they currently did. Those who had used interest- 
based bargaining said the process had distinct advantages when the parties were trying to solve 
complex problems or develop new programs. Yet many reported that the approach did not work 
well for all problems or situations, especially those involving scarce or declining resources. 

Over time a “hybrid” approach emerged in many districts, which included elements of both 
traditional and interest-based bargaining. For example, Columbus’s Johnson said her union’s 
strategy was “kind of like win-win, but not win—win. It’s kind of like traditional bargaining—a 
combination of the two ways of doing it.” The sides identified the issues they would work on and 
then each drafted a proposal on each issue using a formal process that involved a mediator. “So, if 
our issue is discipline, we would draft a proposal, and the administration would have to draft one 
as well.... Everybody drafts proposals, and then we get down to it and we do some traditional 
kinds of bargaining.” 

Other districts used an interest-based strategy to address reform initiatives and a traditional 
approach to reach agreement on salaries and benefits. These presidents argued that interest-based 
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bargaining did not work well to resolve financial disagreements. In Westminster, Colorado, where 
the parties had used certain aspects of interest-based bargaining for 6 years, Lynch said, “Interest- 
based works better with the sidebar issues than it does [with the] financial [issues]. Money issues 
just generally tend to drift to traditional bargaining.” He explained that they always used “separate 
sessions entirely” to discuss monetary and nonmonetary issues. 

Finally, some districts used joint subcommittees both before and during bargaining, thus 
engaging a broader group of stakeholders than the closed-room bargaining sessions of the past. 
Miami-Dade’s Karen Aronowitz explained how negotiators used a set of subcommittees to address 
specific issues, including student achievement, parental involvement, professional development, 
and benefits and compensation. Similarly, Los Angeles’s Duffy said 32 union subcommittees in 
Los Angeles developed proposals that fed into bargaining. Often, while the sides were negotiating, 
joint subcommittees were also meeting, resulting in a process that was far less controlled and 
centralized than traditional, bilateral bargaining. This hybrid approach appeared to be a pragmatic 
strategy enabling the parties to collaborate on parts of the school improvement agenda while 
advocating forcefully on behalf of their constituents. 


A Partnership for Contract Management 


Once a legally binding contract is signed, it must be administered and enforced. In many ways, 
this subsequent process is as critical as—or more important than—collective bargaining to the 
ongoing operations of a school system, because contract language means nothing until it is put 
into practice. 

Virtually all contracts include a grievance process that teachers can use to file a complaint 
about an alleged violation of the agreement. If the grievant is not satisfied with the outcome after 
the administration has ruled, she can usually seek redress from an outside arbitrator. Because 
arbitration introduces additional expense for the union, officials do not pursue every grievance. 
Instead, the union uses an established process to determine which complaints should proceed to 
arbitration. 

The presidents suggested that, although the grievance process continues to have an important 
place in contract management, it is no longer central to labor-management relations. The union 
leaders said that most grievances today are dealt with in routine meetings or through the work 
of committees established to resolve the dispute at the lowest possible level, even before a 
grievance is filed formally. When union representatives decide that a teacher’s complaint is 
legitimate, an officer might call the principal or superintendent directly. For example, in Amherst- 
Pelham, Sheehan said he was often able to “smooth [complaints] over before they become 
a grievance by calling the principal ... and saying, ‘Well, did you know ...?’ And we can 
talk it through.” Sweetwater’s Alex Anguiano described his regular working relationship with 
district administrators: “I have standing meetings [every 2 weeks] with our director of labor 
relations. And a lot of our issues and problems are resolved by these standing meetings that we 
have.” 

In some cases, local contracts include mechanisms for addressing problems before they become 
grievances. For example, the Collier County contract calls for “faculty advisory committees” in 
each school. Jeffers explained, “If there’s a problem at your school, bring it to the faculty advisory 
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committee. They get to set the agenda. They call in the principal and they say, ‘Here are our 
problems; help us fix them.’ And most of the times they get fixed right then.” 

Although presidents often downplayed the role of grievances in day-to-day labor-management 
interactions, many insisted that there were still occasions when the formal process was necessary. 
Collaborative mechanisms worked best when problems could be resolved through conversations 
or changes in behavior. However, if resolution depended on securing more resources (e.g., 
providing substitutes so that teachers would have their guaranteed preparation period) or might 
set an important precedent, the unresolved problem might move through the formal grievance 
process and on to arbitration. 


Collaboration and Continuous Bargaining 


In addition to these efforts to jointly manage the contract, several districts were involved in 
a kind of perpetual bargaining. In today’s dynamic educational environment, pressing issues 
sometimes emerged that required contractual changes and could not wait until the scheduled start 
to formal bargaining. Several districts had created standing committees, composed of teachers 
and administrators, which were authorized to conduct preliminary bargaining about certain issues 
well before the start of contract negotiations. 

Other districts went even further, empowering such committees to amend the contract at any 
time. Colorado Springs’s Valerio explained, “We actually bargain almost all year.” The district had 
a “joint council,” composed of members of the two bargaining teams who met monthly. When we 
asked whether the parties were really bargaining, Valerio responded, “Oh yes.” Some of the issues 
would have to “wait until the contract comes up. Some of them, we do memos of understanding.” 
Often in subsequent negotiations, the parties codified the joint council’s decisions, which had 
already been implemented. Notably, with this process of continuous bargaining, formal contract 
negotiations in Colorado Springs usually lasted only 2 weeks. Similarly, Cullison said that in 
Montgomery County, the “Jabor management collaboration committee ... meets monthly, and 
can actually make changes in the contract.” In 2005-06, for example, the committee revised the 
timetable for teacher transfers well before formal negotiations began. 

Presidents who described successful efforts to broaden the scope of issues and expand the 
timeframe of negotiations stressed the importance of having strong, respectful relationships with 
the superintendent and other district officials. In these settings, union presidents met often with 
superintendents who asked for their views and solicited their help in resolving both difficult and 
routine problems. Some, such as Needham’s Neilsen, relied primarily on scheduled opportunities: 
“There’s a standing monthly meeting between the union officers and all the central administrators.” 
Others, such as Cincinnati’s Sue Taylor, described more fluid exchange with her superintendent: 
“When issues come up, I have her home number. I have her cell phone number. She takes my 
calls, and I take her calls. We often are talking on the phone at 8:00 in the morning before 
meetings start. And there’s very open communication.” Taylor and others agreed that trust is key 
in these relationships, because it allows the parties to openly discuss both traditional concerns 
and innovative ideas. 

The presidents differed in assessing how beneficial collaborative labor-management rela- 
tionships ultimately were to the unions’ interests. Some leaders explained that, despite rely- 


ing on this less adversarial approach, they still resorted to traditional tactics in bargaining or 
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contract management in particularly difficult situations. For example, Pat Santeramo of Broward 
County, Florida, described his response when the superintendent made unilateral decisions that 
compromised teachers’ working conditions: “I just can’t allow that to happen. And we file 
unfair labor practice. And he’ll back off. And we’ll subpoena him. I mean just those things 
that irritate the hell out of somebody; every once in a while you need to kind of poke him in 
line.” However, most presidents reported that it was far better for the president and superinten- 
dent to meet regularly, even when serious disagreements stood between them. Many understood 
that working closely with management did not require abandoning principles or priorities. In 
fact, such interaction was probably the only way to effectively advance the union’s expanded 
agenda. 


SUMMARY AND DISCUSSION 


New realities in the context of public education threaten the future of both teachers unions 
and public schools. Everywhere schools face unprecedented demands for evidence of student 
success under state and federal accountability laws. Some encounter stiff competition from 
charter schools, which have substantially reduced student enrollment and forced districts in states 
such as Ohio to close schools. Districts also struggle to attract and retain a strong teaching force 
as large cohorts of veteran teachers retire and attrition rates among new teachers rise. 

There is wide agreement that the adversarial practices of traditional, industrial unionism are 
not well suited for addressing these complex challenges. In traditional collective bargaining the 
sides are fixed, roles are inflexible, and interactions are ritualized. Reformers intent on making 
schools nimble and competitive would probably not choose collective bargaining as the best or 
quickest route to school reform. However, Moe’s call for eliminating or significantly reducing 
the influence of teacher unions is not substantiated by this study. Teachers’ basic right to organize 
and bargain is codified in 45 state laws and unions are the current reality in most school districts. 
The question is not whether unions will continue to shape local education policy, but how they 
will do so. It is leaders at the local level who will largely determine whether the union obstructs 
or advances school improvement efforts. In districts where the union promotes a professional 
agenda and collaborative relationships with management, its endorsement and influence may 
actually enhance school improvement initiatives. 

As a group, the broad sample of 30 presidents we interviewed did not fit the traditional 
stereotype of hostile labor leaders following the dictates of state and national affiliates. They 
described a working relationship of give-and-take with those organizations rather than one of top- 
down control. Some presidents—typically those who headed larger, more financially independent 
organizations—exercised autonomy in deciding what they would stand for and do. Although 
they might take the affiliates’ positions and priorities into account, they made their own choices. 
Other presidents—often those heading smaller unions—were necessarily more responsive to their 
affiliates. Still, however, most charted their own course. With few exceptions, these presidents 
defined an expanded bargaining agenda, which included some combination of traditional issues 
(wages, hours, and basic working conditions) and reform initiatives (reducing the role of seniority 
in assignments, offering professional development or experimenting with alternative approaches 


to pay). 
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Building a New Culture of Labor Relations 


Many people believe that if problematic provisions are eliminated from contracts, school practice 
will consequently change. However, informal reports from school districts in non-bargaining 
states suggest that certain features of educational practice, which critics attribute to unions— 
for example, reliance on seniority, reluctance to dismiss weak teachers, or rigid application of 
rules—are in fact deeply rooted in school culture everywhere in the United States. Thus, restrictive 
contract provisions are only one part of a larger problem, and a singular focus on reforming or 
restructuring the negotiated agreement will not lead to the needed changes. Those will require 
broader attention to the norms and practices of both unionism and teachers’ work. 

A few presidents’ accounts suggest that the culture of industrial unionism, which is rule-bound 
and adversarial, remains alive and well in their school district. Where industrial practices prevail, 
principals tend to manage their schools by edicts that generate resentment among teachers, or 
they manage defensively, anticipating the union’s challenges and avoiding conflict. Meanwhile, 
union building representatives play their part in a partisan drama, filing charges and threatening 
grievances in an effort to intimidate administrators, even when there are no clear violations of the 
contract. Such administrators and union leaders enact the practices of the old reality, rather than 
adapting to the challenges of the new, which call for different norms of interaction. 

Although a few presidents seemed poised to censure any proposal or action by management, 
most were engaging in new possibilities, standing for unexpected values and shaping their 
larger agenda on behalf of better schools. They often collaborated with management using new 
organizational structures designed to accommodate change. However, moving beyond industrial 
unionism is not easy because it requires changing both rules and culture. Moreover, it demands 
ongoing leadership by both labor and management at all levels of the school district. Local 
presidents are in a key position to lead on behalf of new norms and to promote a new culture of 
labor relations and professional practice through their public statements, selected initiatives, and 
regular interactions with teachers and administrators. 

There is no relationship more important in educational labor relations than that of the union 
president and superintendent. If that relationship fails, little else will work. The presidents’ 
accounts suggested that, for this key relationship to succeed, there must be strength, advocacy, 
and trust on both sides. The trust that enabled presidents and superintendents to move beyond 
conventional labor-management positions was played out in their day-to-day practices—regular 
and reliable meetings, easy access by phone or e-mail, no surprises in the media, and mutual 
respect on public display. 

Some presidents described working in partnership with their superintendent, others told of 
remaining respectful rivals, and several said their superintendent disparaged or ignored them. 
In a very few districts, presidents reported that the superintendent was hired with a mandate to 
break the union and reclaim the rights of management. However, an anti-union agenda on the 
part of management was widely perceived to be antiteacher, which the presidents said not only 
provoked resistance but also increased teachers’ militancy. From the perspective of the presidents 
we interviewed, a management-driven model is not the answer to the challenges schools face 
today, or even an alternative. These presidents may be ready to step up to a newly defined role 
and relationship, but they are not ready to step aside. 

Although the president and the superintendent are influential, they do not hold all the cards in 
the local labor relationship. Ultimately, the power of even the most progressive union president 
or superintendent to build a more productive labor relationship can be threatened by constituents 
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trying to take a more conventional stance in a polarized relationship. On the union side, such 
constituents might include a powerful subgroup of veteran teachers, a longtime executive director, 
an in-house UniServ representative, or members of the executive committee. Sometimes these 
individuals fear that teachers will lose their influence if union leaders relax their grip. On the 
other side, a superintendent may find that he or she cannot ignore school board members who 
worry that students’ interests will be abandoned if the union has too much say. Similarly, the 
public, which expects management to closely control school spending, must be acknowledged. 
Superintendents also must listen to the concerns of principals, who expect their authority to be 
protected. Thus, presidents cannot be seen as being in bed with management and superintendents 
cannot be seen as pandering to the union. 

However, the presidents in this study who worked with school officials to confront the dis- 
trict’s problems did not seem to abandon their members, nor did they avoid conflict as they jointly 
explored possibilities with management. Similarly, the superintendents they described kept stu- 
dents’ interest and the public good clearly in mind while working steadily with the president to 
develop new practices and norms for local labor relations. These productive relationships were 
not said to be smooth or free of disagreements, but they were respectful, intense, and purposeful. 
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TABLE A1 
Study Participants 





Local Union Local School District 





California 

Alex Anguiano Sweetwater Education Association Sweetwater Union High School District 

A.J. Duffy United Teachers Los Angeles Los Angeles Unified School District 

Jim Groth Chula Vista Educators Chula Vista Elementary School District 

Bruce Seaman Grossmont Education Association Grossmont Union High School District 

Sarah Ross Associated Pomona Teachers Pomona Unified School District 
Colorado 

Mark Chavez Boulder Valley Education Association Boulder Valley School District 

Tom Lynch Westminster Education Association Adams 50 School District 

Lori Maag Greeley Education Association Greeley-Evans Weld County 

District 6 

Kim Ursetta Denver Classroom Teachers Association Denver Public Schools 

Irma Valerio Colorado Springs Education Association Colorado Springs School District 11 
Florida 


Karen B. Aronowitz 
Janice D. Brown 


Theo Harris 


Von D. Jeffers 


United Teachers of Dade 

Glades County Classroom Teachers 
Association 

Palm Beach County Classroom Teachers 
Association 

Collier County Education Association 


Patrick A. Santeramo Broward Teachers Union 


Miami-Dade County Public Schools 
Glades County School District 


Palm Beach County School District 
Collier County Public Schools 


Broward County Public Schools 
(Continued on next page) 
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TABLE A1 
Study Participants (Continued) 
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Local Union Local School District 
SSeiaibel sabes thei ee ise ie pai NA es Rk el et 
Maryland 
Gary Brennan Frederick County Teachers Association Frederick County Public Schools 
Bonnie Cullison Montgomery County Education Association Montgomery County Public Schools 
Ann DeLacy Howard County Education Association Howard County Public Schools 
Marietta English The Baltimore Teachers Union Baltimore City Public School System 
Carol Kilby Prince George’s County Educators’ Prince George’s County Public Schools 
Association 
Massachusetts 
Cheryl A. DelSignore Educational Association of Worcester Worcester Public Schools 
Sherrill Neilsen Needham Education Association Needham Public Schools 
Timothy Sheehan Amherst-Pelham Education Association Ambherst-Pelham Public Schools 
Richard Stutman Boston Teachers Union Boston Public Schools 
Paul Toner Cambridge Teachers Association Cambridge Public Schools 
Ohio 
Susan Brooks Mount Healthy Teachers’ Association Mount Healthy City School District 
Melissa Cropper Georgetown Federation of Teachers Georgetown Exempted Village School District 
Rhonda Johnson Columbus Education Association Columbus Public Schools 
Sue Taylor Cincinnati Federation of Teachers Cincinnati Public Schools 


Willie A. Terrell, Jr. Dayton Education Association Dayton City School District 
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Members of organizations traditionally outside public school systems have begun to take on central, 
and in some cases internal, leadership roles and responsibilities in the implementation of ambi- 
tious educational improvement initiatives. How are these arrangements playing out in practice? 
This article explores that question with a 3-year qualitative investigation of the participation of 
two such organizations—a school reform support organization and a consortium of foundations— 
in the implementation of new small autonomous schools initiatives in Oakland (California) and 
Chicago. Using Malen’s politics of implementation framework, this article reveals how external 
organizations influenced the implementation of district initiatives with significant consequences 
for both the reform strategy and their own ability to sustain their involvement in implementation 
over time. The article concludes with implications for the research and practice of educational 
leadership. 


Private and nonprofit organizations, such as philanthropic foundations and school reform orga- 
nizations, typically appear in educational research under the headers “external support providers,” 
“intermediary organizations,” and “nonsystem actors”—as organizations that work with but dis- 
tinctly outside public educational systems (Coburn, 2005; Honig, 2004a). However, the partici- 
pation of these organizations in certain urban districts suggests that not all operate in external, 
supporting, intermediary, or nonsystem roles. 

For example, not only do some private philanthropic foundations make grants to support 
school reform activities conducted by others, but their officers sit as voting members on local 
education reform boards, assume district central office positions, and otherwise take on leader- 
ship roles and responsibilities in implementing ambitious educational improvement initiatives 
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(Gammon, 2007; Hess, 2005). Some nonprofit school reform support providers have assumed 
formal responsibility for running request-for-proposal (RFP) processes and helping create and 
launch new schools (e.g., Davidson, 2002). Such work seems well beyond familiar forms of 
districts contracting out for services and penetrates the core of traditional central office roles 
and responsibilities. These examples suggest that members of some traditionally external orga- 
nizations arguably operate inside public education systems either formally, by taking on specific 
posts within central office subunits and boards, or by otherwise assuming responsibility for 
work fundamental to what central offices do. How, more specifically, are these organizations 
participating in education policy implementation in urban districts? What are the results of their 
participation? 

This article explores these questions with a comparative, qualitative case study of the partic- 
ipation of private and nonprofit organizations in the implementation of new small autonomous 
schools initiatives in Chicago, Illinois, and Oakland, California. These initiatives called on or- 
ganizations formally outside their respective public school systems to assume central leadership 
roles in implementation and therefore offered particularly relevant cases for this inquiry. The 
Chicago High School Redesign Initiative (CHSRI) and Oakland’s Bay Area Coalition for Equi- 
table Schools (BayCES) approached these demands in quite different ways and offered important 
opportunities for contrast. Using Malen’s (2006) politics of implementation framework, I show 
that despite their formal leadership position with respect to the initiative, CHSRI actually op- 
erated in the traditional mode of some private philanthropic foundations—seeking to exchange 
funding, some support, and information for influence and otherwise working to change schools 
and the central office from the outside in. In the end, another reform agenda displaced theirs. 
By contrast, BayCES’s leaders took responsibility for core implementation work that might have 
otherwise been left to leaders within the public school system. Past and present BayCES staff 
also worked within the system in various formal capacities essential to implementation. These in- 
sider strategies fueled school and central office improvements and BayCES’s ongoing influence 
within the district. I discuss why the two organizations operated differently, why their efforts 
yielded different results, and the implications for future educational leadership research and 
practice. 


BACKGROUND 


Brian Rowan offered a fairly comprehensive typology of how organizations outside formal 
school systems participate in school systems at the district-level, mainly through the provision 
of goods and services. He referred to these organizations as part of a “school improvement 
industry” (Rowan, 2002). Participants in this sector include for-profit organizations such as 
textbook publishers and vendors, membership organizations like professional associations, and 
nonprofit organizations including universities and research firms. Rowan argued that school- 
system change dynamics, particularly system resistance to change, partly result from product- 
exchange relationships between school systems and organizations in the school improvement 
industry. For example, textbook publishers’ profit margins partially depend on school systems 
purchasing their materials over multiple years. When school systems select textbooks and develop 
curriculum and professional supports around them, they, too, become invested in maintaining their 
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use in ways that can aid improvement but can also deter reconsideration of such materials in ways 
that may hamper improvement. . 

These and other dynamics of product-exchange relationships may capture how some exter- 
nal organizations and school systems interact. However, other interactions involve not only or 
mainly product exchanges but human exchanges—the participation of employees from these 
organizations in the design and implementation of educational policy initiatives; relationships 
between external organizations and school districts then may entail not only the exchange of 
ideas and funding but also the movement of people into formal central office positions and lines 
of work. These relationships may result from informal agreements, memoranda of understand- 
ing, partnerships and other arrangements different from the fee-for-service contracts typical in 
product-exchange relationships. 

The limited research on external organizations reveals how external organizations operate 
outside public school systems to support others’ work or to advocate for or against particular 
reform agendas mainly carried out by others but not how they take on formal public leadership 
roles inside school systems. For example, I have shown how community organizations can 
enhance various resources essential to central office administrators’ work (Honig, 2004a, 2004b). 
Coburn (2005) elaborated how “nonsystem actors” significantly shape which policy messages 
teachers receive and engage. Marsh (2007) showed how community organizations and citizens 
may increase district knowledge and relationships important to realizing accountability policy 
goals. These studies highlight external organizations’ leadership support functions, including 
resource provision and messaging. However, they do not illuminate the internal participation of 
traditionally external actors. 

This literature does commonly reveal political dimensions of how these organizations operate 
outside school systems, suggesting that a political analysis might provide a useful starting point 
for a fuller conceptualization of their more internal participation. In my own studies, commu- 
nity organizations generated political capital important to district central office administrators’ 
practice. Marsh surfaced political challenges which arise when community partners participate 
centrally in public policy processes. Rowan’s typology, too, reveals political power dynamics of 
the insider—outsider exchange relationships. 

New small autonomous schools initiatives provide a strategic opportunity for exploring the 
potentially new forms of participation of traditionally external organizations. As designed by the 
Bill and Melinda Gates Foundation, these initiatives deliberately designated external organizations 
as the main fiscal agents for these initiatives on behalf of the local public school system, setting 
up those organizations as central initiative leaders. Studying them increased the likelihood that 
I would observe traditionally external organizations operating in internal ways. In 2000, the 
foundation awarded $15.7 million to the nonprofit BayCES to create a regional initiative with an 
initial focus on seeding and supporting new small autonomous schools in the Oakland Unified 
School District (OQUSD). Likewise, in 2001 and 2003, the foundation gave over $20 million to 
Chicago Public Schools (CPS) through CHSRI, a consortium of local funding organizations. 
According to several experts across the country, the Gates Foundation made these awards to these 
“intermediary organizations” rather than public school systems based in part on the assumption 
that their leadership would influence educational improvement more powerfully than school 
districts operating alone or with external organizations’ more peripheral participation. Given 
frequent leadership turnover in some urban districts, positioning external organizations as main 
fiscal agents also promised to increase accountability for the grant funds. 
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CONCEPTUAL FRAMEWORK 


To anchor an exploration of CHSRI’s and BayCES’s participation in implementation, I turned to 
Malen’s politics of implementation framework. This framework derives from research on the pol- 
itics of policy implementation. Because it foregrounds the influence strategies of particular actors 
as consequential to implementation outcomes, this framework helps reveal political dynamics of 
how actors exercise leadership in policy implementation (Malen, 2006; Northouse, 2006). At the 
heart of Malen’s framework, individuals—in this case, members of external organizations—seek 
influence over other actors’ participation in ways that align with their own interests and resources. 
Northouse might distinguish these actors as leaders because they aim to influence others toward 
achieving particular goals. These actors, whether school system insiders or outsiders, influence 
change not necessarily or mainly through their formal positions or mandates, but by whatever 
strategies help them marshal other actors’ support for their priorities. 

Various conditions may mediate the success of external organizations’ influence strategies. 
Other actors’ interests and resources may not align with their interests and prompt those others to 
derail policy goals. If underlying policy premises and “currencies” —the resources associated with 
a policy—are particularly controversial, leaders may face formidable challenges in marshaling 
support for those policies. Institutional and sociocultural conditions such as long-standing political 
struggles for control over education and local social norms also mediate the strengths of these 
influence strategies (see Shipps, 2003; Stone, 2001). Accordingly, implementation results not from 
the sheer power of a given actor or influence strategy but from how actors, policies, influence 
strategies, and contexts interact. 

These interactions unfold over time in fairly predictable patterns called political games (Bar- 
dach, 1977; Firestone, 1989; Malen, 2006). For example, the politics of policy dilution results 
when actors’ influence strategies fail to align with policy premises and other actors’ interests and 
resources, and when dimensions of the sociocultural and institutional context do not support such 
alignment. The politics of policy amplification may occur when actors do achieve this alignment. 
The games concept prompts researchers to explore how influence strategies play out over time 
and contribute to predominant political reform dynamics. 

This framework focused my inquiry on the following specific research questions: Who are 
the main actors in implementation and what are their interests and resources? What main influ- 
ence strategies do they employ over time? What outcomes seem associated with their influence 
strategies? How do actors, policies, influence strategies, and dimensions of their implementation 
contexts interact in ways that mediate how reform dynamics play out over time? 


METHODS 


I explored these dynamics with data from a broader comparative qualitative case study of new 
small autonomous schools initiatives in CPS and OUSD that captured the early years of implemen- 
tation in each district.! The two districts offered important opportunities for contrast as elaborated 
in the Findings section. This article draws on 112 interviews with 78 respondents and evidence 





\{ conducted field work between 2000-03 in OUSD and 2003-06 in CPS. For an elaboration of the research design 
and methods, see Honig (2009). 


398 M. I. HONIG 


from meetings convened to support implementation. Interviews probed how actors participated 
in implementation and conditions that mediated actors’ participation. CPS meeting data included 
minutes from CHSRI Board meetings between January 7, 2002, and July 17, 2005. In OUSD I 
directly observed school board hearings, central office staff meetings, and other implementation 
meetings. I also conducted extensive reviews of documents in each district including local news- 
paper articles, policy documents, e-mail correspondence, and Listserv postings. I wrote detailed 
field notes to highlight key observable dimensions of the implementation context in each district. 

Lanalyzed these data using NVivo software in several phases. First, I coded for basic dimensions 
of implementation including: policy design features, actors’ participation in implementation, 
outcomes, and conditions that helped or hindered implementation as well as for data type. 
Subsequently, I selected Malen’s framework to guide this subanalysis and refined my coding 
with codes derived from her framework. For example, within data coded as “participation,” I 
distinguished activities consistent with “influence strategies.” I ultimately reported those strategies 
that I could verify with at least three different types of data sources (i.e., interviews with three 
different respondents or a combination of interviews, observations, and documents). Likewise, I 
reduced the data on “outcomes” to those I could associate with specific influence strategies using 
at least three different data sources. 


FINDINGS 


Although CHSRI and BayCES were similarly positioned as fiscal agents and formal leaders, their 
actual influence strategies and outcomes differed greatly. CHSRI’s participation was consistent 
with the traditional outsider influence strategies of some private philanthropic foundations— 
seeking to trade funding, support, and information for influence. Although successful in other 
arenas, in this case, such influence strategies did not achieve many of the central office or 
school changes CHSRI sought. In the end, CPS absorbed CHSRI schools into another initiative, 
spearheaded by CPS and the Chicago business community, which differed in some significant 
ways from CHSRI. Such dynamics reflect the politics of policy dilution—processes by which lead 
actors fail to align other actors’ interests and resources with their own and with policy premises 
and currencies. By contrast, BayCES employees took on work and formal positions that would 
otherwise have fallen to district employees. These insider influence strategies seemed to fuel 
positive school and central office changes that weathered the school district’s takeover by the State 
of California. BayCES’s influence grew even as district administrators came to participate more 
centrally in implementation.” Such dynamics reflect the politics of policy amplification, which 
reveals “how education systems and their surrounding communities .. . develop the governance 
structures and the cross-sector coalitions ...that install and sustain major education reforms, 
particularly in urban centers” (Malen, 2006). For an outline of influence strategies in each 
district, see Table 1. 


These claims relate to findings up until 2003 in OUSD and 2006 in CPS. They do not necessarily reflect the current 
state of implementation in these districts. 
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TABLE 1 
Comparative Influence Strategies 
School-Level Influence Strategies Central Office-Level Influence Strategies 
CHSRI e Providing grants and developing e Providing information 
grant-related requirements - About schools 
e Coaching support mainly tied to grants - To CPS executives 
BayCES e Providing school incubation and e Developing, modeling, and transferring the Small 
ongoing coaching Schools Incubator as a model of central office 
operations 


e Designing and helping implement a model of 
central office transformation 


Note. CHSRI = Chicago High School Redesign Initiative; CPS = Chicago Public Schools; BayCES = Bay Area 
Coalition for Equitable Schools. 


CHICAGO: THE POLITICS OF POLICY DILUTION 


In this section I argue that CHSRI’s main influence strategies only occasionally leveraged their 
intended central office- and school-level changes. In the end, its initiative was displaced by 
another in ways that significantly dampened CHSRI’s ongoing influence. 


Actors: Interests and Resources 


CHSRI Board members were mainly local foundation officers who, in one member’s words, 
“bought seats at the table by contributing to the Gates Foundation match”—the requirement that 
local districts match Gates Foundation grant funding. Other CHSRI voting members included 
the Gates Foundation, the Chicago Teachers Union, the Chicago Mayor’s Office,’ and CPS.* The 
president and chief executive officer of the Chicago Community Trust and the chief education 
officer for CPS co-chaired CHSRI. CHSRI staff included an executive director with a reputation 
for significant expertise in community and youth development. Even so, many respondents 
overwhelmingly associated CHSRI with Chicago-based foundations. For example, most CPS 
administrators in interviews repeatedly referred to CHSRI and its board as “the foundations.” 

According to documents and interviews, CHSRI aimed to improve students’ academic achieve- 
ment, increase graduation rates, and prepare students for postsecondary opportunities. To this end, 
CHSRI planned to create upwards of 30 small schools by converting some large comprehensive 
high schools into multiple small autonomous schools and, beginning in 2003, by creating new 
stand-alone schools called “new starts.” 


3A review of meeting minutes from the study period suggests that the mayor’s office was a limited participant in 
CHSRI—not appearing on the meeting attendance list in 2002 or after February 2003. In January 2005, they were dropped 
from the attendance list as someone whose absence was noted. 

4The document “Update of CPS and CHSRI RFP Process,” dated May 17, 2004, provides an historical overview of 
the CPS/CHSRI RFP process. This document specifies that voting members of the initiative board have included the CPS 
chief executive and academic officers, the president of the Chicago School Board, the mayor’s office (with two votes), 
and the Chicago Teachers Union, in addition to local funders and the Gates Foundation. 
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At the outset of the initiative, CHSRI’s resources included funding from the Gates Foundation’s 
initial $12 million grant as well as the local foundation match. One CPS central office administrator 
commented that CPS’s total contribution to the initiative significantly dwarfed CHSRI’s but that 
CHSRI’s contribution was still “larger than any other fund currently being contributed to the 
Chicago Public Schools.” This individual commented that‘CHSRI resources also included “all 
the very important players in the city, [people who are powerful] not just for the money they 
control [now] but for future money.” Two other central office administrators reported that CHSRI 
also brought “the political buffer or the political engine that is formed by the foundations— 
the national [Gates] foundation and its drivers, and then the locally-based foundations and its 
drivers—that both catalyzes the system to change and helps to cover the system in times of trouble 
during the change.” 


Main Influence Strategies 


To realize their goals, CHSRI aimed to influence schools directly. They also worked to change 
CPS central office policies and practices to support such school changes. 


School-Level Influence Strategies. Study data overwhelmingly suggested that CHSRI 
sought influence over school-level change using requirements tied to school grants as a main 
lever. CHSRI Board members and staff agreed that the several stages of their grant application 
process provided multiple formal and informal opportunities to interact with and intervene in 
school decision-making processes to improve school-level implementation. Interviews with CPS 
and CHSRI staff as well as CHSRI Board minutes and other documents revealed that as part of 
their application process, CHSRI staff visited prospective applicants, assessed their readiness to 
participate, and reported results to the CHSRI Board. After the RFP was issued, CHSRI required 
schools to submit evidence of their readiness including letters of intent, the application itself, and 
typically, multiple budget justifications and revisions. CHSRI frequently made a school’s receipt 
of funding contingent on their satisfactory response to these reporting requirements, as well 
as multiple rounds of other questions. CHSRI Board meeting minutes and member interviews 
suggest that one full application cycle consumed between one half and two thirds of their meeting 
time each year and was their single most time-consuming agenda item. Respondents’ comments 
also supported my claim that CHSRI’s main school-level influence strategy involved requirements 
tied to funding. For example, one respondent reflected, “I think they see themselves very much 
more like a traditional funder who says ‘I got a pot of money, here, do your proposal.’ ” 

CHSRI Board members and staff also intended coaching support to influence school-level 
changes. Both CPS- and CHSRI-affiliated respondents confirmed that CHSRI staff routinely 
counseled participating principals and required school-based teams to participate in an action re- 
search project. CHSRI also contracted with other organizations such as WestEd and Northwestern 
University to run workshops for schools on specific issues. However, these and other respondents 
commented that these areas of work were modest, especially early in implementation. Because 
the CHSRI Board often required schools to accept this support as a condition of receiving CHSRI 
funding, I viewed it as a component of CHSRI’s grant-related influence strategies. 
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Central-Office Influence Strategies. CHSRI aimed to leverage central office change pri- 
marily by generating information about school-level implementation and sharing it with CPS 
members of the CHSRI Board through board meetings and formal reports. Their main information 
sources included an independent evaluation of the initiative involving a series of school snapshots, 
a qualitative analysis of professional learning communities within participating schools, and a 
mixed-methods analysis of student and school outcomes. Official meeting minutes of the CHSRI 
Board in 2002 indicated that CHSRI’s Evaluation Subcommittee operated “to ensure that the local 
evaluation will help the Advisory [i.e., CHSRI] Board, the school system, and the participating 
schools. It is important that we get lessons learned that can be spread to other schools.” Three 
CHSRI Board members explained that using evaluation as a central office influence strategy 
stemmed from board members’ assumptions that its “real-time” information about participating 
schools could influence central office decision makers. 

CHSRI Board meetings also almost always included progress reports on prospective or par- 
ticipating schools, created by CHSRI staff visits to schools and the schools’ written responses 
to board members’ questions. As one board member described these reports, “They... . give the 
CHSRI Board ...a sense of ... how [school] people were doing; [CHSRI staff] are ... doing 
all the day-to-day work and then summarizing what the [schools’] strengths and weaknesses 
are.” Discussions about school reports frequently generated questions and/or ideas that founda- 
tion participants asked board members from CPS to consider. For example, in early 2002, the 
CHSRI Board asked CPS to develop a template for an accountability agreement with participating 
schools that specifically addressed such questions as, “Should schools be given a time limit to 
improve?” 

CHSRI’s approach to using information as a central office influence strategy reflected two 
underlying assumptions. The chief executive’s office and others in what one respondent referred 
to as the “upper echelons” of the central office should be the primary audience for CHSRI’s 
information, and if CPS executives had access to school-level information, they would be will- 
ing and able to respond in ways that reinforced CHSRI’s agenda, including increasing CPS 
investments in the participating schools. As partial evidence of the first claim about the main 
audience for the reports, the CPS chief executive officer and chief academic officer were the 
only voting members from the school system on the CHSRI Board, and board meetings were the 
main forums where the evaluators presented their reports. Staff of the CPS new small schools 
office responsible for managing implementation, several Area Instructional Officers to whom 
the CHSRI school principals reported, and the CHSRI evaluators all confirmed in interviews 
that CPS staff outside the executive office had little to no input on the evaluations’ design and 
were not a main audience for the reports even though they were heavily involved with those 
school. 

Regarding the second assumption, the evaluation reports focused almost exclusively on school- 
level implementation issues and tangentially used principals’ reports to highlight how the central 
office impeded implementation. When asked about this school-level focus of the evaluations, one 
CHSRI Board member commented, “I hadn’t thought of this before you asked. But there is an 
operating assumption that if [central office executive staff] learn about what is going on at the 
schools then they will know what to do at the central office level. As opposed to analyzing, at 
the central office, what they are doing that may be of support or hindrance to what these schools 
could or should be... . If there are real needs at the central office [for an analysis of the central 
office] then the central office has to put a mirror onto itself.” 
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Outcomes in Chicago 


Arguably, the school application process worked as an influence strategy to expand the CHSRI 
initiative to an increasing number of schools. Between 2002 and 2006, 19 schools opened under 
the CHSRI banner, and by 2006 the CHSRI Board had specific plans to support an additional 9 
schools. 

However, the CHSRI grants and grant-related requirements were weak levers of school im- 
provement and, perhaps, counterproductive. CHSRI staff members readily noted such limitations 
and attributed them in part to CHSRI’s initial inexperience with school support. In the words 
of one, “Every time we get ready to release a [Request for Proposals] we start doing [technical 
assistance]. We back it out a little bit further and even then I think we have some limitations in 
terms of the amount of stuff that we have to offer... . I think this is kind of on the minimal end 
and I think it needs to be further developed.” 

CPS central office respondents most closely involved with CHSRI were generally positive 
about CHSRI’s attempts to leverage school improvement through their grant processes. In the 
words of one, CHSRI has “made about as strong an effort as they could.” Others in the CPS central 
office and outside CPS offered more critical assessments. In the words of one, CHSRI’s approach 
to helping schools develop and implement school improvement plans “is to run workshops that 
pull a few teachers out and take them downtown on topics they have decided in advance without 
consulting or visiting with schools to determine what the schools really need... . They seem to 
have no real understanding that that doesn’t get down into the classrooms at all.” 

Others indicated that the sheer number of reports CHSRI required negatively affected schools. 
“The principals feel like they are constantly being asked to do more reports, that there is no 
trust... . [The reports] are another set of obligations that distracts you from the very intense work 
of getting the school up and running properly... . I find a lot of times . . . they [school principals] 
are too overwhelmed . . . but they say, ‘Hey, that is where the money is coming from. I had better 
salute””” Another elaborated that the limitations of CHSRI’s school-level influence strategies 
stemmed from their approach that “all of their schools have to [participate in workshops as a 
condition of receiving funding] regardless of whether it works for that particular school. It’s all 
pull-out downtown; they are not in teachers’ classrooms helping them.” 

Regarding central office change, multiple respondents reported that CHSRI helped advance a 
focus within the CPS central office on small high schools. One characterized CHSRI as a “catalyst. 
_.. I think they catalyze the process of changing education by making people think differently.” 
Another added, “In terms of being concerned and involved, I would give them [CHSRI] an 
A....In the last few years I have never heard Chicago talk about education in the way they are 
talking ... now.” 

CHSRI’s information-based influence strategies by many accounts leveraged at least two 
significant central office policy changes. For one, respondents unanimously attributed the creation 
of a separate Area Instructional Officer (central office administrators to whom all school principals 
report) for CHSRI schools to the information and attendant pressure generated by the CHSRI 
evaluation and extended discussions at CHSRI Board meetings. Advocates argued that a dedicated 
Area Instructional Officer would help ensure CHSRI principals received support and evaluations 
from someone well versed in their particular challenges and goals. Several CPS respondents also 
credited the timing and ultimate drafting of a revision of the school board’s small autonomous 
schools policy to ideas and pressure generated by CHSRI Board members. 
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However, the CHSRI Board more typically raised issues—sometimes over many years—that 
did not result in CPS policy or practice changes. For example, at multiple meetings over two 
years the CHSRI Board discussed the importance of changing CPS enrollment policy for CHSRI 
schools from 600 students to no more than 500 students in keeping with initiative goals and 
CHSRI interests to shrink school size. Yet despite frequent formal and informal discussions, CPS 
leaders, citing budgetary concerns, ultimately decided not to lower the cap (Honig, 2009). 

Although CHSRI was powerful enough for the CPS Chief Executive Officer (CEO) to consult 
with them about decisions such as appointing school principals, such consultations did not 
constitute evidence of CHSRI’s influence over those decisions. For example, in 2004 CPS decided 
to phase out freshman enrollment at two high schools that they ultimately planned to restructure as 
new small autonomous schools. Someone from the chief executive office contacted CHSRI staff 
before issuing a press release to ask if it was okay to say they would become part of the initiative 
even though the CHSRI Board had not yet voted on those schools. One respondent explained that 
such a call was a courtesy but not really necessary: “As a CEO, he can say whatever he wants.” 

Attendance at CHSRI meetings provides another indicator of CHSRI’s limited influence on 
the CPS central office. A review of the 46 attendance rosters available for the period between 
January 2002 and July 2005 revealed that although the CHSRI co-chair from the philanthropic 
community attended 42 meetings, the CHSRI co-chair from the CPS CEO’s office attended 
only 27 meetings. Nor did the co-chair from CPS actually chair the four meetings that the other 
co-chair missed. The CEO of CPS attended only 10 of 46 meetings—most of them in 2002. At 
almost one third of them (7 = 13) no voting member from CPS was present. 

Approximately 2 years into the CHSRI effort, leaders from CPS, the Chicago business com- 
munity, and the Chicago mayor’s office launched the Renaissance 2010 initiative to create 100 
new schools by 2010, including charter schools that did not fit CHSRI’s priorities. Shortly 
thereafter, board minutes and interviews with CPS staff indicated CPS would include all new 
schools, including CHSRI’s, under Renaissance 2010. 

In sum, the evidence suggests that the trajectory of CHSRI’s participation in implementation 
reflects the politics of policy dilution game. In this version, the CHSRI Board and staff members 
exerted minimal influence over the actions of the CPS central office and its schools. All respon- 
dents interviewed about the potential significance of Renaissance 2010 for CHSRI reported that 
Renaissance 2010 signaled a significantly diminished leadership role for the CHSRI. In fact, in 
2008, the CHSRI Board dissolved. 


OAKLAND: THE POLITICS OF POLICY AMPLIFICATION 
The dynamics of BayCES’s participation in the new small autonomous schools initiative un- 
folded as the politics of policy amplification—the process by which cross-cutting coalitions 
come together to grow and sustain major education reforms. In this subsection, I describe these 
dynamics. 
Actors: Interests and Resources 


Unlike CHSRI that launched specifically to support implementation of the Gates Foundation 
grant, BayCES began 10 years earlier than the start of their Gates-funded small schools initiative 
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as a regional office of the Coalition for Essential Schools (Bay Area Coalition for Equitable 
Schools, 2007). Their formal mission statement suggests BayCES’s long-term interests included 
fostering dramatic improvements in “educational experiences, outcomes, and life options for 
students and families who have been historically underserved by their schools and districts. We 
do this by teaching, coaching and supporting school, district and community leaders to create and 
sustain systems of equitable and high performing small schools.” BayCES’s executive director 
described BayCES’s interests in the new small autonomous schools initiative at an Oakland 
School Board hearing in 2003 in this way: “Our mission is to create and sustain a network of 
high achieving and equitable schools.” 

BayCES’s starting resources included access to the Gates Foundation grant and the local 
match but also an established cadre of school reform support coaches whom they deployed 
mainly to OUSD schools. In addition, various data sources including other researchers’ reports 
(e.g., Davidson, 2002; Mediratta et al., 2004; Mediratta, Shah, & McAlister, 2008) suggest 
BayCES also benefitted from formal and informal relationships with two other organizations, 
Oakland Community Organization (OCO) and OUSD, specifically to support implementation. 
OCO, a coalition of faith-based institutions, began organizing around the creation of new small 
autonomous schools in the mid-1990s. In 1998, OCO invited BayCES to partner with them in 
what OCO leaders called a “small schools movement.” OCO leaders reported that they sought 
out BayCES as a main partner because they realized that OCO had high capacity to organize 
communities but not the ability themselves to help schools provide powerful and equitable 
teaching and learning. As one OCO leader described, “[OCO] organize people to rise up and take 
power back from systems that oppress them. Historically in many flatland [low-income] Oakland 
neighborhoods, that has meant the schools. But at some point when you are organizing people to 
create new schools inside the system, you need to work inside the system. That’s where BayCES 
comes in.” In 1998, OCO, BayCES, and OUSD formed the Small Schools Working Group to 
examine growing and supporting new small autonomous schools. Subsequently, in 1999 OUSD 
created a position of director of Small Schools and invited BayCES to help draft a new small 
autonomous schools policy which the Oakland School Board approved in 2000. BayCES then 
received $15.7 million from the Gates Foundation to support new small autonomous schools of 
various kinds. 


Main Influence Strategies 


BayCES’s influence strategies, like CHSRI’s, focused on both schools and the central office. 
However, the nature of these activities departed starkly from CHSRI’s: CHSRI’s school-level 
influence strategies mainly involved grants and grant-related requirements, whereas BayCES’s 
entailed more intensive school-by-school supports; CHSRI’s central office influence strategies 
focused on making school-level information available mainly to executive CPS staff, whereas 
BayCES staff moved into the central office to design and help implement central office reform 
strategies that promised to help implementation. 


School-Level Influence Strategies. BayCES invested a significant amount of time and 
other resources in designing and running the “Oakland Small Schools Incubator” in which 
OUSD required prospective and funded school teams to participate. The incubator, “supports 
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teams of family members, teachers, students, and others as they design new small autonomous 
schools, respond to the RFP, and open and operate their new schools” (OUSD, 2002, np). Various 
respondents’ reports and observations confirmed that the Incubator led teams, usually comprised 
of teachers and community members, through a process of establishing a vision for a new school 
and building relationships and a culture to support that vision. As one central office administrator 
described, ““BayCES has a coaching mission. It is ... pedagogical. It is relational. It’s about 
community visioning around what people want school to be for their children and the children 
they are teaching. And they have built the Incubator to engage schools in those processes before 
they officially apply for a small schools grant to help them get ready. And they wrap the Incubator 
around them after they receive the grant and beyond to help extend the work.” A school principal 
confirmed, “BayCES was there when we walked in the door [to start our school] and we went 
through the Incubator process and here on the other end they are here—in fact they are here [at 
our school] right now working with our ... team.” 


Central-Office Influence Strategies. BayCES also used the Incubator as a strategy for 
influencing central office change, specifically to model for central office administrators how 
to support school implementation. Some respondents called the Incubator a “school system 
incubator.’ As the BayCES executive director explained, “So the Incubator, the way we are 
working with people, the creating of these alliances at the grassroots level is part of what we are 
trying to do to build a base [for reform] where we take accountability out of the big macro-system 
and put it into a local relationship. Still with the district being a major player but a new kind of 
player.” 

Multiple OUSD central office administrators reported that they understood BayCES would 
ultimately, in the words of one, “turn over more and more of the work as we [in the central office] 
came on line to take it on.” As another explained, “Like we say, this reform is ‘community driven, 
BayCES supported, and district owned.’ BayCES has had a very significant role in being sort 
of the proxy for the district, so although they do a lot of that development it was being done 
in proxy for the district. So the Incubator was always a partnership with the district, BayCES, 
OCO.... [Early on] they [BayCES] took the lead on areas of curriculum related to the design of 
schools and what school designs should be but it is a partnership and those roles should change 
over time.” 

Beyond the Incubator, BayCES also aimed to influence the central office by developing a 
model of a transformed central office that would better support implementation and by moving 
into formal central office leadership positions to help implement that model. As one school 
principal explained, BayCES, “just decided that there is no way to fix the district [central office] 
so they went this shadow district route . . . to create a district that is outside of the district that goes 
through an incubator process that models [i.e., resembles] the process that schools go through. 
Eventually, that becomes a district and the current district will get phased out.” 

While developing their model for broader central office reform, BayCES’s leaders generated 
ideas and proposals for how OUSD might change specific central office policies to support im- 
plementation. For example, in 2003 when OUSD central office administrators were updating 
their school facilities master plan, BayCES commissioned a local researcher to generate rec- 
ommendations for a state assembly bill to enable the new schools to occupy certain buildings 
temporarily while their permanent sites were constructed or retrofitted. The BayCES executive 
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director and an OUSD central office administrator, who is a former BayCES staff person, helped 
an assistant superintendent actually incorporate those ideas into the facilities master plan. Shortly 
thereafter, the BayCES executive director formally remained a BayCES employee but dedicated 
his professional time to work within the central office on developing and implementing their 
broader central office transformation plan. . 


Outcomes in Oakland 


During my period of study, nine new small autonomous schools opened in OUSD, and almost as 
many were participating in the Incubator to prepare a future application. Between 2001 and 2003, 
OUSD central office administrators increasingly participated in the Incubator. For example, when 
asked in 2003 to what extent OUSD’s participation in the Incubator had changed or remained the 
same in the preceding 2 years, one central office administrator reported, “I think we have got a 
much better idea of what the steps are to incubating a school. So we [central office administrators] 
added the implementation plan. That is a huge step and we had a major role in that. That was 
our curriculum. That was all our department people so that was a big step of ownership that the 
district took for this Incubator piece.” School leaders confirmed that central office administrators 
increased their involvement in the Incubator, and, in tandem, the quality of the support they 
provided to participating schools. 

In 2003, the new small autonomous schools initiative in Oakland faced a formidable threat to 
its sustainability: The state of California dissolved the local school board, removed the superinten- 
dent who championed the initiative, and placed a state administrator in charge of all school district 
operations. These developments could have fueled the dilution of the initiative, especially given 
that the state takeover stemmed from concerns about OUSD’s fiscal management and that at the 
time the cost effectiveness of the initiative was disputed in local and national forums. Instead, the 
number of participating schools grew under state oversight. Many Oakland actors advocated for 
continuing the initiative so it is not possible to credit BayCES specifically with these outcomes. 
However, as an indicator of BayCES’s contribution to this outcome, it was the new state admin- 
istrator who ultimately invited the BayCES executive director to work within the central office 
on their central office transformation plans. By many accounts, including observations from my 
ongoing research in OUSD, BayCES’s ideas about central office transformation became Expect 
Success!, a major, multimillion dollar initiative to recreate nearly all central office functions to 
support principles consistent with the new small autonomous schools initiative (OUSD, 2007). 

In sum, these dynamics reflect the politics of policy amplification. They demonstrate how 
over time BayCES’s influence helped leverage deeper school and central office engagement in 
implementation. BayCES-originated ideas served as the blueprint for broader, ongoing central 
office reforms and, by the end of my data collection period, the BayCES executive director and 
another former BayCES staff person had taken on significant lines of work within the central 
office to advance the initiative goals to which BayCES was fundamentally committed. 


COMPARATIVE DYNAMICS OF THE IMPLEMENTATION GAMES 


Why did the games differ in each district? I found that differences between the games stem 
from several specific conditions. First, the politics of implementation framework amplifies the 
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importance of the match or mismatch between CHSRI’s and BayCES’s interests and resources, 
their influence strategies, other actors’ (viz., in this case, school district leaders’) interests and 
resources, and policy premises and currencies. CHSRI mainly relied on traditional institutional 
tools of some private philanthropic foundations as their main influence strategies: grants and 
grant-related requirements and generating information about but not becoming deeply involved 
themselves in implementation. Although successful in other contexts these tools weakly matched 
the CHSRI initiatives’ policy demands and CPS central office leaders’ interests and resources. In 
this vein, two CHSRI Board members explicitly reported that they viewed the CHSRI Board as a 
policy advisory board, not a deeper participant in implementation; even so, they did not structure 
their meetings to follow up on key policy issues in ways that might have deepened their influence. 
Many respondents acknowledged that CHSRI staff attempted to provide various forms of school 
and central office assistance but that their small size and relative inexperience—specifically 
with classroom teaching and learning, and especially in the high need schools CHSRI selected 
early in implementation—meant they were a particularly limited resource in this policy context. 
Likewise, the information that CHSRI generated, although potentially useful, generally did not 
address questions that central office leaders raised as important to their decision making. For 
example, CHSRI’s evaluation focused on generating school-level information, but CHSRI Board 
minutes and interviews with central office administrators suggested executive central office staff 
also sought information about central office systems—such as information about the impact of 
the initiative on the overall central office budget and about central office policies that interfered 
with school autonomy. 

By contrast, BayCES promised to make such important, specific contributions to implementa- 
tion that another small schools movement leader, OCO, tapped BayCES to participate centrally in 
implementation. BayCES’s influence strategies took them inside the central office as main leaders 
of particular major lines of implementation work generally within the purview of central office 
employees. BayCES chose strategies that both complemented the capacity of their partners and 
that shifted over time as the central office increased its capacity to support implementation. Thus, 
BayCES’s leaders seemed intentionally to forge the kinds of matches over time that the politics 
of education framework highlights as particularly consequential to implementation success. 

Second, dimensions of the institutional context in which the two organizations operated— 
namely, their organizational histories and past practices—seemed particularly important influ- 
ences on these implementation games. CHSRI formed anew specifically to manage the Gates 
Foundation grant. CHSRI’s individual foundation members and staff brought various forms of 
expertise to the CHSRI Board, but CHSRI as an organization had no past experiences on which 
to draw. Especially given the limited examples or role models of external organizations in such 
central educational leadership capacities, CHSRI members may have been particularly inclined 
to rely on influence strategies they knew well from their work as individual foundations in other 
situations, even though those strategies were not a particularly good match in the present case 
(Barley, 1986; March, 1994). 

BayCES, too, may have engaged in implementation in ways consistent with their histories and 
prior practices. But in BayCES’s case, their history and practice were on a trajectory that seemed 
well matched for the new small autonomous schools initiative. By the time they received their 
first Gates Foundation grant, BayCES had already led school reform efforts for many years. Many 
Oakland respondents recounted that BayCES, OCO, and other Oakland leaders had been building 
a movement specifically around new small autonomous schools long before Gates Foundation 
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funding became available and that such a history of joint work may have generated relationships 
and momentum important to implementation. 

Three, the dynamics of policy design likely mediated these implementation games. Especially 
when debates about policy designs are contentious, actors participating in those debates will tend 
not to reconcile their conflicts or make fundamental decisions especially about policy winners and 
losers. Rather, actors will pass those challenges and tensions on to implementation in ways that can 
interfere with implementation progress (Ingram & Schneider, 1991; Malen, 2006). In Oakland, 
BayCES experienced little conflict over their selection as the fiscal agent for the Gates grant. At 
the time of policy design, agreement about how various external partners would participate in 
implementation was so strong that central office leaders and school board members codified the 
division of labor between BayCES, OCO, and OUSD in the first new small autonomous schools 
policy and in a formal Memorandum of Understanding. 

By contrast, in Chicago, significant tensions arose around policy design, particularly around 
the selection of CHSRI as the fiscal agent for the Gates Foundation grant; such tensions may have 
set the stage for an especially challenging implementation experience for CHSRI. Multiple re- 
spondents recounted that when it launched its small schools grant program, the Gates Foundation 
initiated conversations with various long-standing Chicago-based organizations about serving as 
their fiscal agent. These respondents reported that members of the foundation community argued 
that Chicago lacked, in the words of one, “an appropriate intermediary for you [the Gates Foun- 
dation] to place your dollars and we will form an advisory board. You can place the dollars in a 
donor-advised fund. That is how we will match the funds but we want a significant amount of 
control over the dollars given that they are going to be a huge amount and there was no strong 
intermediary to step up and take the dollars.” The appointment of CHSRI to lead the initiative 
in such circumstances likely fueled significant counterproductive conflicts among potential part- 
ners in implementation and thereby curbed CHSRI’s influence during implementation. As partial 
evidence of such conflicts, multiple respondents associated with CHSRI and these other organi- 
zations accused each other of handpicking the schools they work with, not having an interest in 
strengthening all schools, and not having the capacity to support schools well. 

The participation of other organizations in implementation as part of the institutional context 
in which CHSRI and BayCES operated also seemed to shape the implementation games. Many 
Oakland respondents, including BayCES staff, reported that BayCES’s work hinged significantly 
on OCO’s efforts to organize communities in support of small schools. The Oakland Cross City 
Campaign for Urban School Reform reinforced BayCES’s central office transformation plans by 
linking OUSD leaders to their counterparts in another district that had similarly reformed their 
central offices. Private philanthropic foundations in Oakland contributed a match for the Gates 
Foundation grant but largely deferred to the partnership between BayCES, OCO, and OUSD 
when it came to the strategic direction of the initiative. 

The involvement of so many actors could have created unproductive conflict and diffused 
BayCES’s influence. To the contrary, these organizations over time formed a coalition around 
common initiative goals and consensus about how various actors would participate in implemen- 
tation. As a central office administrator commented, 


Our second year . . . the pressure and relationships in the partnership [between BayCES, OCO, and 
OUSD] became more—it wasn’t clearly defined. It was murky. It was difficult. We were stepping on 
each other’s toes. OCO went, “More, more, more” and we were like, “We can’t, we can’t, stop!” It 
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was difficult. ... This year we have done some . .. mending and repairing of the partnership. I think it 
has to do with the fact that we all have learned and accepted that . . . [there are] certain places and in 
certain spaces where our partners can be and others [where they] can’t be. Also, our [Memorandum 
of Understanding] that we’ve developed defining the responsibilities—that has also helped us know 
what we can partner around and what we can’t. And the way we wrote the MOU is we are clear about 
what’s what but it can grow and change as we do. 


The prior discussion about the dynamics of policy design suggests a far different and less 
collaborative relationship between CHSRI and other school reform support organizations. 
Furthermore, a iack of clarity among members of those organizations, including CPS, may have 
impeded others’ willingness to support CHSRI’s leadership. For example, when asked what role 
CHSRI plays in implementation, respondents’ comments varied widely and included “Holds 
CPS accountable to make sure that the funds are directed and not misdirected”; “Does not create 
schools. It gives money to support new schools that are created by the central office”; “I don’t 
know, to be honest with you, I don’t know. I think right now they approve grants that come 
or proposals”; Their role “is not around providing dollars or special programs”; “They have 
provided advice on the transition of some large schools to smaller schools and I think they also 
have been champions of the schools”; “Partners, cheerleaders ... [who] bring expertise to the 
table’; and “Advocates for the schools but beyond that I’m really not sure.” Two CHSRI board 
members reflected that they never developed a formal memorandum of understanding with the 
school district and that in retrospect some agreement about the roles and responsibilities for 
various organizations in implementation could have advanced their work. 

As another part of the institutional context in both districts, the Gates Foundation itself par- 
ticipated in implementation in ways that seemed consequential to the reform dynamics described 
here. For instance, partway through my study period, the Gates Foundation changed its grant- 
making strategy from almost exclusively funding single intermediary organizations within each 
district to funding other organizations including schools and school district central offices. Gates 
Foundation funding to Oakland continued to flow through BayCES, further augmenting their 
resources. By contrast, their grants to Chicago grew to include individual schools, community- 
based organizations, and ultimately the Renaissance 2010 Board. Such contributions fueled the 
resources of other actors in ways that likely dampened CHSRI’s influence. 


CONCLUSIONS AND IMPLICATIONS 


This article aimed to build knowledge about the participation of traditionally external organiza- 
tions in education policy implementation. New small autonomous schools initiatives provided 
an important opportunity for this exploration in part because these initiatives explicitly called 
on organizations outside public school district systems to spearhead these ambitious change 
strategies in their respective districts. Previous research on these types of organizations suggested 
that political dynamics, although by no means the only dimensions of implementation, might 
be particularly salient to the experiences of these organizations. Malen’s politics of education 
framework provided a basic set of concepts to anchor an initial analysis of their participation in 
implementation. 
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I show how CHSRI’s and BayCES’s influence strategies played out in CPS and OUSD as two 
quite different implementation games: the politics of policy dilution and the politics of policy am- 
plification respectively. In brief, CHSRI chose influence strategies that positioned them as school 
system outsiders trying to leverage changes in other people’s work. Their actions seemed to re- 
flect relationships between school systems and external partners that hinge on product-exchange 
relationships—in this case, mainly the exchange of funding and certain kinds of information for in- 
fluence. Their influence strategies in light of policy demands, other actors’ interests and resources, 
and contextual conditions operated as weak levers for advancing their goals. By contrast, BayCES 
members took on formal and informal roles and responsibilities in implementation that in many 
ways positioned them as school system insiders. BayCES participated in implementation—not in 
product-exchange, but in human terms—by moving in and out of the central office themselves to 
take on significant roles and responsibilities, and by forging various relationships over time that 
were important to implementation. 

The two implementation games seemed to depend on the following: the match or mismatch 
among the organizations’ chosen influence strategies, other actors’ (viz., school district leaders’ ) 
interests and resources, and policy premises and currencies; dimensions of their institutional 
contexts including their professional and organizational histories; dynamics of policy design, and 
the participation of other organizations. In this view, the focal organizations’ influence strategies 
were not objectively more or less powerful but a better or worse fit under their circumstances. 

This study, like other strategic case studies, is limited in its power for generalizing to prac- 
tice. Nonetheless, this study raises a number of issues and questions that educational leaders 
might productively engage. For one, these cases suggest that under some circumstances expand- 
ing the traditional scope of external organizations’ participation in implementation may buoy 
implementation. Arguably, initiatives such as new small autonomous schools create such signifi- 
cant challenges for schools and central office systems that they demand not only funding, funding 
contingencies, and information related to others’ work, but also intensive, hands-on assistance 
with the daily challenges of implementation from able partners (Gallucci, Boatright, Lysne, & 
Swinnerton, 2006; Honig & Ikemoto, 2008). Such initiatives may benefit significantly from ex- 
ternal organizations operating more like insiders—actors who take on roles and responsibilities 
related to specific lines of work—rather than solely or mainly functioning as outside advocates of 
how others should participate in implementation. Such expanded participation for external orga- 
nizations does not necessarily mean central offices or schools lose authority or the ability to exer- 
cise leadership themselves. Such participation can actually help build leadership capacity within 
schools and central offices for increasing their own participation in implementation over time. 

To help ground decisions about their participation, leaders of external organizations might 
review the lessons of these cases and explore: To what extent are our interests and resources lined 
up with those of other actors we aim to influence and various dimensions of the contexts in which 
we operate? How can we analyze such relationships to improve our strategic planning around our 
participation in implementation? 

The success of ambitious educational improvement initiatives often depends on the extent 
to which schools and central offices break from long-standing institutional patterns and engage 
in new work practices. This study suggests that external organizations too may find that long- 
standing institutional patterns of their own practice are a better or worse fit with policy demands 
and contexts. Such institutional shifts can take time, and, as the BayCES example suggests, 
may hinge on the accumulation of human, social and political capital over many years. Leaders 
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of these organizations might consider: How can we make decisions about our participation in 
implementation to address particular implementation needs but in ways that strategically build 
on our own current capacity? How can we view our participation in implementation not as a 
fixed set of activities but as part of a dynamic change process? How can we build agreements and 
processes into our own work so that our participation continues to evolve as our capacity and our 
partners’ capacity changes and grows? 

These cases also raise important questions for school-systems leaders to consider in helping 
them participate productively in their partnerships with external organizations. Among those 
questions: How can we fuel relationships with various external organizations that help deepen our 
collective capacity for implementation? As both cases demonstrate, in positive and negative ways, 
no one actor—not even a consortium of foundations or a well-established school-reform support 
provider—had all or arguably even most of the resources necessary for the implementation of 
policies as ambitious and complex as new small autonomous schools initiatives. BayCES’s success 
seemed to hinge significantly on recognition of these interdependencies by the main reform 
partners and the creation of formal agreements around those interdependencies. Furthermore, the 
evolution of their agreements over time too seemed consequential to their positive implementation 
outcomes. How can school-system leaders forge relationships that build capacity inside and 
outside formal school systems to fuel educational improvement goals? 

This analysis also has a number of implications for research on educational leadership. First, 
this analysis highlights the importance of examining and building knowledge about how tradition- 
ally external actors participate in increasingly central educational leadership roles. The featured 
cases suggest that under certain circumstances the more internal participation of these external 
actors may be important for realizing ambitious educational improvement goals. Future research 
might explore cases where these external actors take on other kinds of increasingly internal 
roles. For example, my research ended as BayCES’s executive director increased the span of 
his responsibilities within the central office while remaining a BayCES employee. How do such 
arrangements play out over time? What challenges or opportunities arise as external organizations 
expand their internal participation? What are the positive and negative consequences of having 
private or nonprofit sector employees carry out main lines of work within public systems? 

Second, the politics of implementation framework provided one basic set of concepts for 
organizing a description of how these organizations participated in implementation in these dis- 
tricts but the framework is limited in a variety of respects that future research might productively 
address. For example, the framework amplifies actors’ “influence strategies.” However, other 
frameworks, particularly those more explicitly rooted in theories of leadership, might reveal a 
broader range of activities or other ways of framing the activities in which external organizations 
engage. This study confirmed the importance of framing external organizations’ participation in 
implementation in human rather than product-exchange terms. This framing amplifies the impor- 
tance of the day-to-day experiences of individuals in implementation and how their relationships 
matter to implementation. What more specifically are those day-to-day experiences? How do they 
vary within and between organizations participating in implementation? How do those individual 
experiences matter to the inter-organizational dynamics revealed here? 

Third, future research should aim to build knowledge about these new insider—outsider dynam- 
ics in the context of other policy and district cases. As the politics of implementation framework 
highlights, the resources and currencies of a focal policy matters substantially to how implementa- 
tion unfolds. Insider strategies pursued by established organizations like BayCES in contexts like 
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Oakland’s may be particularly promising given the ambitious scope of the new small autonomous 
schools initiatives. However, external organizations might do well to pursue traditional outsider 
influence strategies in the context of other policy initiatives or other districts. As researchers 
accumulate findings in different policy and district contexts, analyses across those cases might 
explore which influence strategies appear productive or powerful across them and which seem 
particular to certain policies and places (Honig, 2006). This accumulation and analysis of multiple 
cases might subsequently allow analysts in the future to predict how political games are likely to 
play out in other settings. 

More broadly, this study is one in a growing line of research that challenges scholars of 
educational leadership to expand the traditional scope of our own field from examining mainly 
school-level leadership to truly focusing on educational leadership—the pursuit of influence over 
educational improvement from a variety of organizational and institutional positions both inside 
and outside formal school systems. The practice of educational leadership seems to be moving in 
this direction. What will it take for more scholars of educational leadership to break from our own 
longstanding institutional patterns and more regularly expand our conceptions of what counts as 
educational leadership? 
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This article presents a conceptual framework derived from institutional theory in sociology that offers 
two competing policy contexts in which charter schools operate—a bureaucratic frame versus a 
decentralized frame. An analysis of evolving charter school types based on three underlying theories 
of action is considered. As charter school leaders pursue new and different forms of schooling, 
they are challenged by well-established bureaucratic rules and norms that define what it means 
to be a legitimate school. Microlevel institution-building efforts, however, may represent viable 
alternatives to the institutional order of public education as charter school leaders attempt to achieve 
scale. 


This article presents a conceptual frame rooted in new institutional theory that aims to discover 
the challenges of creating, sustaining, and expanding charter school organizations as they evolve 
within the wider institutional environment of public schools. The utility of institutional theory 
as a lens for studying charter school reform lies in its ability to explain the complex relationship 
between decentralized schools and their environments. Institutional theory emphasizes the influ- 
ence that an organization’s cultural environment has on organizational structure and behavior, and 
it seeks to understand the ways in which cultural rules from the environment shape or constrain 
organizational action (Scott, 2003). 

In the context of schools, institutional theory explains how school organizations are respon- 
sive to the demands of their institutional environment, largely because their formal structure is 
“loosely coupled” from their technical activities—specifically, there is a lack of coordination 
and control over the technical processes linked to classroom instruction (J. W. Meyer & Rowan, 
1977, 1978; J. W. Meyer, Scott, & Deal, 1983). Institutional theory thus challenges rational 
conceptions of organizations as internally goal driven and technically efficient. Instead, institu- 
tional theorists argue that legitimacy is derived from conformity to the normatively held rules 
and scripts of the institutional environment, rather than instructional effectiveness; for schools, 
organizational success thus depends on factors other than technical efficiency (J. W. Meyer et al., 
1983). 
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Presumably, the decentralized and autonomous nature of charter schools allows them to 
operate free from bureaucratic controls to challenge and redefine institutional classifications.! 
Yet ironically, the challenge for charter schools to create new definitions of schooling may be 
nearly insurmountable, considering that the very institutionalized scripts that have resulted from 
bureaucratic controls are those that in the wider institutional environment define both effective 
and legitimate schooling. In essence, as charter schools evolve, not only are they challenging 
the tenets of the institutional environment which they are part of, but their evolution and the 
development of new alternative forms of schooling is challenged by the rules, norms, and scripts 
of the more established and robust institutional environment that has come to define and dominate 
legitimate forms of schooling. Whether charter schools can in fact overcome these institutional 
challenges to pursue new and alternative models of schooling remains to be seen. 

Absent from existing research are studies that examine the challenges and constraints that 
decentralized school organizations face when evolving within the wider institutional environment 
of public schooling and the efforts by charter organizations to overcome them. Although charter 
schools are granted autonomy from many regulations that govern traditional public schools, they 
are not evolving within a decentralized policy vacuum that insulates them from the forces of the 
wider institutional environment. Rather, charter schools are still subject to normative definitions 
of effective schooling advanced by the institutional environment. 

The conceptual framework presented in this article aims to fill this void in the research 
by assisting us in both predicting and identifying challenges that decentralized schools will 
confront while evolving in the wider context of an institutional environment. The frame- 
work is grounded in the following empirical question: Will charter schools increase their 
prospects of organizational survival by organizing around the tasks of technical production, 
as opposed to responding to the institutionalized rituals that define legitimate schooling in the © 
institutional environment? The framework presents two competing conceptual contexts in which 
charter schools operate—a bureaucratic frame versus a decentralized frame (Huerta, 2007; Huerta 
& Zuckerman, 2007). On one hand, charter schools are decentralized from bureaucratic over- 
sight so that they may focus on their internal technical goals and local accountability demands 
(decentralized frame), but on the other hand, charter school organizations are evolving within 
a robust institutional environment that expects conformity to normative institutional definitions 
of schooling (bureaucratic frame). Although charter schools, in theory, should be free to pursue 
new and different forms of schooling that challenge long-standing institutionalized patterns of 
teaching and learning, they are at the same time constrained by taken-for-granted institutional 
rules that have come to define legitimate schooling in the United States. 

The framework draws from new institutional theory that emerged in the 1970s led by John 
Meyer and colleagues and incorporates recent developments in new institutional theory by re- 
sponding to literature that calls for new analysis of the changing environment in education 
(described as “an evolving technical environment’) through the lens of institutional theory (Boyd 
& Crowson, 2002; H. D. Meyer & Rowan, 2006a; Rowan, 2006). Rowan (2006) explained how 


! When ritual classifications (e.g., teacher certification, instructional time, class size, curriculum, instruction, resource 
allotments, organizational structure) become institutionalized and accepted as the normative definition of effective and 
legitimate schooling, then educational organizations are inclined to adopt such classifications into their formal structure 
to derive resources and legitimacy from the wider institutional environment (J. W. Meyer & Rowan, 1977, 1978; Rowan, 
2006; Rowan & Miskel, 1999). 
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both the new institutionalism in organizational sociology and the American education system 
have evolved considerably since the 1970s. Thus, the changing context demands a revisit of how 
we might apply new institutional constructs that can account for both an evolving theory and its 
application to changing policy contexts, including the “tightening control of government over the 
core technology of schooling” linked to standards and accountability reforms (H. D. Meyer & 
Rowan, 2006b, p. 8); the growth of market-based reforms, which “now seem central to education 
reform” (Rowan, 2006); and the pluralistic interests that have led to changing power structures in 
education, including community control, that may lead to more diffuse and decentralized forms of 
schooling (Bidwell, 2006; H. D. Meyer, 2006; Rowan, 2006). Last, the framework incorporates 
microinstitutional perspectives that have also evolved since the late 1970s, including agency- 
based approaches to institutional change that reveal more about the process of institutionalization 
by focusing on actors’ microlevel responses to coercive isomorphic? pressures of the institutional 
environment (Burch, 2007; Spillane & Burch, 2006; Zucker, 1991). 

We begin by first considering how different theories of action linked to the decentralized 
charter school reform have resulted in diverse forms of schooling. We categorize the theories of 
action into three distinct forms: diffuse forms of leadership and school organizations that have 
evolved in the form of community-based schools, external management of schools that are tightly 
aligned with market-based principals and are promoted by educational management organization 
(EMO)-run charters, and newly emerging charter management organization (CMO) charters that 
have created hybrid management structures that both mimic existing institutional governance 
structures and adopt centralized efficiencies as tools to scale-up more technically oriented forms 
of schooling. Then we present our theoretical framework for analysis of decentralized school 
organizations derived from new institutional theory. We posit that as charters evolve, they are 
forced to balance their internal objectives linked to technical efficiency with bureaucratic de- 
mands linked to institutional conformity (J. W. Meyer & Rowan, 1977; J. W. Meyer et al., 1983). 
Finally, we conclude with a discussion that contextualizes the challenges that charter schools 
encounter in their attempts to achieve scale. In doing so, we discuss how attempts by charter 
organizations to engage in local-level institution building and create new definitions of effec- 
tive schooling are challenged by the demands of the wider institutional environment of public 
education. 


COMPETING THEORIES OF ACTION 


In this section we discuss three distinct theories of action—what we call local control, market 
forces, and management recentralization—that reflect the assumptions about educational change 





2Tsomorphism is an important concept that addresses how organizations that are responsive to an institutional environ- 
ment evolve and interact within an institutional environment that defines or prescribes the normatively accepted practices 
of organizations. The institutional view of isomorphism is developed by Powell and DiMaggio (1991), who identified the 
following three mechanisms of institutional isomorphic change: (a) Coercive isomorphism results when organizations 
are persuaded, forced, or coerced to change by other organizations which they are dependent on. Coercive isomorphism 
is most common in organizations that operate in legal and political environments. (b) Mimetic isomorphism is linked to 
organizational uncertainty or ambiguity, which results in the imitation or mimicking of legitimate organizational struc- 
tures in the institutional environment. (c) Normative isomorphism is based on the norms of professionalization where 
organizations adapt to, or reflect the increased specialization or professionalization in a field to signal legitimacy. 
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advanced by charter school reform leaders. We present them in static form that allows for little 
overlap between them; however, in practice many of the charter movement’s reform leaders might 
subscribe to the tenets of more than one, creating a dynamic between the competing theories 
of action that is not always captured in the upcoming sections. Emphasizing the distinctions 
between charter schools is important for analytic purposes and for understanding the emergence 
of differentiated charter school types within the broader institutional context of public education 
(Brown, Henig, Lacireno-Paquet, & Holyoke, 2004). 

Like other reform efforts, the charter school movement contains multiple theories of action 
that have reflected different, sometimes competing, theories of action held by its major actors 
over time. Despite these differences, researchers have tended to treat charter schools as an 
“undifferentiated mass,” with little regard for the values and assumptions that give rise to particular 
types of charter schools (Henig, Holyoke, Brown, & Lacireno-Paquet, 2005, p. 514). However, 
some recent studies have begun to differentiate among charter school types, primarily those 
affiliated with for-profit EMOs and those that have their roots in the nonprofit, public service 
sector (Brown et al., 2004; Bulkley, 2002; Bulkley & Hicks, 2003; Henig et al., 2005). Henig 
et al., for example, introduced a theoretically based typology of charter schools that organizes 
charter school founders into two groups—‘market” versus “mission” driven reformers. To study 
differences in charter school behavior, they further disaggregated these categories into subtypes, 
such as charter schools founded by educators, parent and community groups, or social service 
organizations, each with their own distinct theory of action for how to provide better educational 
outcomes for their students. These theories of action tell a mixed story of both divergence and 
convergence in organizational behavior among charter schools of different types and highlight 
the distinctiveness of charter school founders and the different theories of action they hold about 
educational change and upon which they build their schools. 


Local Control 


When state education laws opened the door for local control in the form of charter schools in the 
early and mid-1990s, the first founders were teachers and leaders of groups with a shared sense 
of missionary zeal and altruistic purpose, including community-based organizations, parents, 
and social service agencies (Henig et al., 2005; Fuller, 2000). They opened schools with very 
specific missions and visions that connected to particular pedagogical themes or particular types 
of students. In Washington, DC, for example, most of the first wave of schools to open after 
passage of the District of Columbia Public School Reform Act of 1995, also known as the 
Charter School Act, did so under the auspices of smaller community-based organizations, such 
as the Latin American Youth Center or Associates for Renewal in Education (Henderson, 2001), 
seeking autonomy to exercise what they believed was in the best interest of their students. 
These early charter school operators defined the problem of public education in the same terms 
as did Albert Shanker, the former influential head of the American Federation of Teachers who first 
introduced the charter concept in the late 1980s. “The problem,” according to Shanker (1988), “is 
that the traditional school structure virtually prohibits the teacher from arranging alternate ways 
the student might learn” (p. 92). According to this theory of action, students and teachers remain 
“locked” into a rigid schedule that emphasizes the “assembly-line processing of children” (p. 95) 
while a bloated bureaucracy increases in size year after year. The solution, or action, then, is for 
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teachers or activist groups to design and implement their own instructional programs according to 
their locally defined conceptions of good schooling. Broad-scale education reform would result 
from the divergent and hopefully innovative school programs educators would develop with their 
newfound autonomy to do something new and different in schools. 

Adherents to this theory of action tend to come from two camps. The first, educational 
professionals, seek greater autonomy and are usually motivated by a particular pedagogical 
vision, which they cannot realize in the traditional public school system (Henig et al., 2005). 
The second, grassroots community organizations, seek greater control over the education of 
their constituent students. Henderson (2001), for example, observed that groups of parents or 
neighborhood leaders often see charter schools as “the logical response to a school system that 
has not responded to its community” (p. 32). Bruce Fuller and his colleagues (Fuller, Gawlik, 
Gonzales, & Park, 2004) go one step further, suggesting that charter activists “have become the 
new cultural relativists” (p. 96), creating schools that advance a specific cultural agenda, local 
identity, and particular way of raising children. 

For both groups, qualified and motivated principals and teachers, who share similar pedagog- 
ical visions and who can create learning communities according to local definitions of fairness 
and effectiveness, are key assets in this theory of action. The movement’s early years relied 
on principals and teachers who were “willing to put in extremely long hours to work out the 
inevitable kinks in a new program, recruit families and teachers to a school with no track record, 
even mortgage their own homes to create their dream school” (Lake, 2007, p. 2). Opening new 
charter schools is similar to starting a new business; it requires a particular kind of educational 
entrepreneur who can manage the many facets of school start-up—instruction, operations, tech- 
nology, finance, human resources, and real estate (Hassel, 2003). Consequently, this theory of 
action depends heavily on teachers’ and principals’ sense of mission and purpose to build schools 
according to their own visions for education. 


Market Forces 


Quite apart from local control, market forces is a theory of action that depends on competition 
and corporate efficiency as the key to better education. Adherents to this theory of action define 
the problems of public education in terms of an overly politicized, monopolistic government 
bureaucracy that creates inefficiency and inequity in the system (Chubb & Moe, 1990; Nathan, 
1996). The focus in this theory of action is on economic principles of free market systems that 
emphasize parents as consumers who will seek to maximize their self-interest in pursuing their 
children’s education. Charter schools are thought to compete with one another for students in an 
educational market that determines which schools succeed and fail based on consumer choice 
and decision making. By creating new incentive structures, competition forces charter schools 
to improve their performance as they attempt to meet the needs of their clients (C. Lubienksi, 
2001). 

Proponents of this theory of action, like those of local control, believe that market forces 
will drive broad-scale education reform by encouraging innovation through autonomous charter 
schools: however, the driving force behind innovation is very different under the two theories of 
action. Whereas chartering opens the door for teachers and parents with innovative ideas to open 
new schools under the local control approach, market principles of choice may spur innovation 
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as schools search for the best ways to improve performance and create efficiencies as a result 
of competition with one another for students (C. Lubienksi, 2003). This theory of action thus 
positions schools as businesses that operate according to private-sector patterns of organizational 
behavior within the public domain of education (Brown et al., 2004; Henig et al., 2005; C. 
Lubienski, 2001, 2003, 2005). 

Although smaller “Mom and Pop” charter schools may in fact ascribe to the tenets of market 
forces, typically larger, for-profit EMOs are the operators closest aligned to this theory of action 
(Brown et al., 2004, p. 1039). EMOs are private companies that take responsibility for the 
management of some or all of a charter school’s operations and that explicitly pursue a profit 
motive (Bulkley, 2002; Scott, 2009). EMOs were a late entry to the charter school movement 
compared to the early “Mom and Pops,” having focused their attention on realizing profits in the 
district-run public schools prior to turning their attention to the charter school market (National 
Charter School Research Project, 2007; Wilson, 2005). But EMOs quickly realized the potential 
of charter schools operating in a decentralized context, accompanied by the removal of unwieldy 
bureaucratic controls found in a traditional school (Wilson, 2005). In the late 1990s and early 
21st century, the number of EMOs entering the charter sector and the number of charter schools 
they operated increased dramatically, nearly quadrupling in size between 1998 and 2004 (Molnar, 
Wilson, & Allen, 2004). 

There are some discernible differences between the ways EMO-operated charters behave 
compared to other charter schools. Achieving economies of scale is a key asset in the market 
forces theory of action, so not surprisingly EMO-founded charter schools tend to be larger 
than other charter schools (Brown et al., 2004; Henig et al., 2005), and take steps to cut costs, 
such as reportedly serving fewer students with special needs (Miron & Nelson, 2002). They 
also grant less autonomy to school-level decision making than intended by original charter school 
reformers (Brown et al., 2004; Bulkley, 2002) and are more likely to engage in high-cost marketing 
campaigns to recruit students (Henig et al., 2005), an attribute associated with the behavior of 
private sector businesses. 

Like any good charter school, EMO-driven schools require principals and teachers committed 
to the vision of the school. Fidelity to the school model is critical for organizational success, yet 
it has proved challenging when EMOs enter multiple communities, each with their own locally 
conceived definitions of public schooling. To overcome this challenge, some EMOs have de- 
signed their own leadership development programs, such as the Edison Leadership Development 
Academy, to train principals in their particular model. They have also initiated broad-based, 
national recruitment campaigns for teachers and principals, and established human resource 
management practices aimed at improving the pipeline of teachers and administrators trained in 
the specific principles of the EMO’s model (National Charter School Research Project, 2007). 
Strategic human resource management is one of many practices that EMOs have borrowed from 
the business sector under the market forces theory of action, attempting to make EMOs more 
competitive in the high-stakes environment of charter school education. 


Management Recentralization 


Management recentralization is the term we use to describe the theory of action that corresponds 
with the newest wave of charter school reform, the emergence of nonprofit CMOs. CMOs are 
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centrally directed nonprofits that manage networks of schools, usually in one region or one state, 
utilizing common instructional models and school designs to achieve consistency and alignment 
across their schools (New Schools Venture Fund, 2006; Wilson, 2005). By organizing centrally, 
CMOs can leverage their size and resources to help charter school principals and teachers 
overcome the entrepreneurial challenges of building and sustaining new schools in the complex 
environment of public education. 

The term CMO originally stems from an effort by the NewSchools Venture Fund, a venture 
philanthropy spun off from the venture capital firm Kleiner, Perkins, Caufield, and Byers, to 
distinguish nonprofit charter management organizations from their for-profit EMO counterparts 
(Wilson, 2005). Although on the surface CMOs appear similar to for-profit EMOs, they embody a 
different theory of action. Adherents to the management recentralization theory of action believe 
the pace of charter school reform has been too slow and has yet to achieve enough high-quality 
schools to have an impact on the broader system of public education. They see that the original 
vision for charter school growth—teachers and community groups starting new, innovative, and 
ultimately better schools under the local control theory of action—has been too difficult to realize 
(Harvey & Rainey, 2006). Although studies of charter school achievement have yielded mixed 
findings, it has become increasingly clear that charter schools, on average, do not outperform 
traditional public schools in any significant way (e.g., Carnoy, Jacobsen, Mishel, & Rothstein, 
2005; Hoxby, 2004a, 2004b; S. T. Lubienski & C. Lubienski, 2006). Moreover, recent research 
around charter school innovation reveals greater instructional and curricular conformity among 
charter schools than originally predicted by both the local control and market forces theories of 
action (C. Lubienski, 2003, 2004). After 15 years of reform efforts, charter school leaders have 
begun to rethink their approach to educational change. 

To achieve high-quality charter schools at scale, many charter school leaders have adopted a 
new approach that emphasizes centrally “managed growth” (National Charter School Research 
Project, 2007, p. 9) as a strategy to achieve broad-scale education reform. Successful charter 
school founders, who opened their schools under the local control theory of action, now seek to 
replicate their success by opening up additional schools. For example, Amistad Academy is a 
highly regarded charter middle school started in 1999 by a group of Yale Law students in New 
Haven, Connecticut, that eventually grew into a leading charter management organization. In 
2003, the Amistad founders launched Achievement First, a nonprofit CMO designed to take the 
Amistad model to scale. They organized a central office to manage the expansion effort, which 
ensures fidelity to the original Amistad school model by creating alignment around multiple areas, 
such as curriculum, recruitment, and talent development. Today, Achievement First manages 15 
charter schools in Connecticut and New York, all modeled after the original Amistad Academy 
(Achievement First, 2009). 

Supporters of management recentralization tend to be leaders of established, successful char- 
ter schools who want to expand their operations, as well as the movement’s intellectual leaders 
who recognize that the “frontier era” of charter school growth, in which unprepared and un- 
dercapitalized charter school founders were once welcome, is over (National Charter School 
Research Project, 2007, p. 9). Instead, they focus now on creating institutions that can bring 
successful charter school models to scale. Doing so requires central planning and coordination, 
the reverse for a movement aimed at decentralizing public education. The key resource in this 
theory of action is the CMO structure—decentralized vis-a-vis the broader system of public 
education, yet tightly linked internally by a standard school model and standard set of operating 
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practices—which provides charter leaders with the organizational structure to build not just one 
school, but an entire charter school system. 

Achieving scale, however, has to date presented charter school reformers with many challenges, 
an issue which we take up in the final section. In the next section, we present a theoretical 
framework that helps explain the challenges charter schools face when addressing rule-based 
demands linked to institutional legitimacy while attempting to advance new and innovative forms 
of schooling that are more attuned to the technical environment of local schools. 


THE INSTITUTIONAL AND TECHNICAL ENVIRONMENTS 
OF EDUCATIONAL ORGANIZATIONS 


Looking at decentralized school policy through the lens of institutional theory allows us to 
identify whether the decentralized conditions under which charter schools operate can influence 
changes within the wider institutional environment. Principally, it is necessary to focus on how 
decentralization might influence institutional shifts that could accommodate new innovations 
created by the charter school movement, or whether the existing norms, rules, and expectations 
of the institutional environment lead decentralized schools to adopt the accepted normative 
definitions of effectiveness that are linked to bureaucratic and rule-based compliance. 

An institutional perspective assumes that charter schools, like all public schools, are still 
operating within a highly institutionalized environment. The institutional environment identifies 
legitimate forms of schooling via a school’s conformity to environmental conditions that define 
effectiveness through standardization and certification procedures (the symbolic rituals, norms, 
and myths of the institutional environment), rather than measurable outcomes (J. W. Meyer & 
Rowan, 1978). By concentrating on coordinating ritual classifications (e.g., teacher certification, 
instructional time, class size, etc.) rather then the technical aspects of education (e.g., measurable 
outcomes), schools can avoid controlling their outputs and avoid inspection, which can call 
attention to the lack of coordination between instruction and outputs. 

However, recent scholarship in institutional theory suggests that the nature of early institu- 
tional analysis has not kept pace with new advances in education policy and reform specifically, 
policy shifts that may demand schools to operate in a much more elaborate technical environment 
than they did when Meyer and Rowan first developed the constructs that identified educational 
organizations in the late 1970s and 1980s. For example, Rowan and Miskel (Rowan, 2006; Rowan 
& Miskel, 1999) argued that institution building on the part of state and federal government agen- 
cies, education professionals, and private organizations has advanced an increasingly technical 
theory of education along with the capacity to inspect and monitor instructional performance in 
schools and classrooms. 

That institutional environments can change over time is consistent with earlier work in in- 
stitutional theory. Scott and Meyer (1991), for example, explained that technologies can be 
introduced into institutional environments and become institutionalized in their own right, adding 
to the technical nature of a field. But they argued that the coexistence of institutional and techni- 
cal environments will lead to high levels of internal conflict, as organizations seek to operate in 
accordance with both strong technical and institutional controls. According to Rowan and Miskel 
(1999), this appears to be the case today in public education, “where a lack of consensus over 
appropriate institutional arrangements for schooling is clearly on the rise” (p. 365). 
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We posit that charter schools represent prime examples of the conflict that arises when or- 
ganizations evolve in the context of institutional and technical demands, trapping them between 
technical pressures for performance and institutional pressures for conformity. In essence, shift- 
ing policy demands that call for increased technical efficiency may lead to an exchange of the 
primary sources of legitimacy from institutional to technical definitions of schooling. For exam- 
ple, schools today are expected to be held accountable first for outputs such as making adequate 
yearly progress stipulated by the No Child Left Behind law, yet they also are accountable for 
ensuring that bureaucratic inputs such as standardized curricula are in place. However, it is not 
clear the extent to which recent policy shifts have resulted in redefining formal sources of legiti- 
macy centered on technical efficiency or whether traditional institutional expectations remain the 
primary source of legitimacy for schools. 

In the context of decentralized charter school reform, schools will have to make significant 
inroads in redefining normative criteria of effective schools, both at the local level and in the 
wider environment, to successfully challenge the existing and robust institutionalized defini- 
tions of effective schooling. However, institutional theory predicts that “organizations fail when 
they deviate from the prescriptions of institutionalizing myths: Quite apart from technical ef- 
ficiency, organizations which innovate in important structural ways bear considerable costs in 
legitimacy” (J. W. Meyer & Rowan, 1977, p. 353). In other words, organizations that chal- 
lenge institutionalized conceptions of effective schooling may be seen as ineffective and risk 
their legitimacy. Organizing around the technical core in attempts to sustain innovations—as 
charter schools in theory are supposed to do—while challenging the rituals and belief struc- 
tures of the institutional environment may in time prove to be the biggest challenge for new 
charter school organizations seeking legitimacy while operating in a highly robust institutional 
environment. 

Even more so than traditional public schools, charter school survival may depend on both con- 
formity to institutional norms and technical efficiency (Huerta, 2007). The exchange for freedom 
from the bureaucracy that a charter school enjoys comes with the provision of performance ac- 
countability. Thus, in principle, charter schools will be more likely than traditional public schools 
to challenge the loose coupling norm of education by organizing around their technical core. 
Charter authorizers, for example, can revoke a school’s charter for failure to meet the stated and 
agreed-upon objectives for student performance, leading to school closure, presumably situating 
charter schools in even stronger technical environments than those of most traditional public 
schools. Yet of the 436 charter schools that have closed nationwide in the last 16 years, only 9% 
have closed because of academic achievement issues, whereas 91% have closed for financial or 
organizational mismanagement and other reasons that are associated with rule-based compliance 
linked to upholding institutional norms, not technical efficiency (Center for Education Reform, 
2006). Thus, as charter school reform evolves, there is evidence that legitimacy, and ultimately 
organizational survival, still depends on upholding traditional institutional definitions of effective 
schooling. 

Without normatively acceptable legitimated accounts of their existence, charter organizations 
will be seen as negligent or irrational by external constituencies, thereby strengthening internal 
dissidents and jeopardizing an organization’s survival (J. W. Meyer & Rowan, 1977). Whether 
charter schools can sustain organizational structures that are responsive to their technical envi- 
ronments, while still operating within the wider institutional environment, is the principal theme 
that drives this analytical discussion. 
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TABLE 1 
The Competing Environments of Charter Schools 
Bureaucratic Frame Decentralized Frame 
(Institutional Environment) (Technical Environment) 
Organizational Schools are responsive to bureaucratic Schools operate in a decentralized context and 
Environment demands and attuned to a well-defined are more attuned to their technical 
institutional environment, which results in environment, resulting in tighter control of 
tighter coordination of ritual technical activities and less organizational 
classifications and less organizational differentiation (J. W. Meyer, Scott, & Deal, 
ambiguity (J. W. Meyer, Scott, & Strang, 1983; Pfeffer & Salancik, 1978). 
1987; Scott, 1998). 
Organizational Schools are open systems, closely linked to Schools are closed systems and structured 
boundaries the institutionalized definitions of around the technical core. They decouple 
production. They rely on ceremony in the technical work from “environmental 
form of certification and standardization conditions,” which could disrupt production, 
to decouple their technical core from their resulting in a tighter link between goals of the 
formal structure (J. W. Meyer & Rowan, organization and the technical core (J. W. 
1978; J. W. Meyer, Scott, & Deal, 1983). Meyer, Scott, & Deal, 1983). 
Organizational Schools are subject to institutionalized rules Schools that are free from regulatory controls 
structure and norms and are organized around a set and subject to locally defined demands will 
of ritual classifications that define create innovative organizational structures, 
legitimate forms of schooling (J. W. teaching and learning methodologies, and 
Meyer & Rowan, 1978; J. W. Meyer, local accountability measures. 
Scott, Strang, & Creighton, 1988). 

Accountability Effectiveness is measured by the ability of a Effectiveness is measured by meeting explicit 
school organization to respond and outcome goals set by local actors, as opposed 
conform to rule-based bureaucratic to bureaucratic conformity and rule-based 
definitions of efficiency (J. W. Meyer, accountability advanced by the institutional 
Scott, & Deal, 1983). environment (J. W. Meyer, Scott, & Deal, 

1983). 


i 


Competing Charter School Environments 


Table | provides a detailed description of the organizational context advanced by the institutional 
and technical environments of public schools. For the purposes of this analytical discussion, 
we divide the organizational environments defined by institutional theory into two conceptual 
frames—the bureaucratic frame and the decentralized frame. These two conceptual frames pro- 
vide a richer context to the organizational structure and institutional pressures that schools must 
navigate. Based on the assumptions advanced by the wider theory of action of charter schools, 
where effectiveness is defined by locally conceived measures of accountability rather than bu- 
reaucratic demands, we describe the decentralized framework of charter schools as a closed 
system where schools are structured around the processes linked to their technical environment. 
In contrast, according to the teachings of institutional theory, traditional public schools operate 
within a bureaucratic framework and operate as open systems that are responsive to rule-based 
accountability advanced by the institutional environment. 


Organizational Environment. Institutional theory describes school organizations as open 
systems that are reliant on the external institutional environment for organizational survival. 
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However, an organization that substitutes the institutional environment for a new technical en- 
vironment in essence becomes a closed system that is more attuned to its technical core of 
production (J. W. Meyer et al., 1983). J. W. Meyer et al. (1983) explained how “the technical 
organization faces in toward its technical core and turns its back on the environment, while the 
institutionalized organization turns its back on its technical core to concentrate on conforming 
to its institutional environment” (p. 47). In the context of charter schools, organizing around the 
technical core means that a more explicit link is made between teaching and learning practices and 
expected measurable outcomes, and legitimacy is no longer derived from upholding rule-based 
accountability. 


Organizational Boundaries. As closed systems, charter schools will seal their technical 
core from institutional demands and allow the school organization, via locally created definitions 
of effectiveness, to manage production (J. W. Meyer et al., 1983). However, as J. W. Meyer and 
Rowan (1977) explain, “organizations which have structural elements not institutionalized in their 
environments should be more likely to fail, as such unauthorized complexities must be justified 
by claims of efficiency and effectiveness” (p. 361). In other words, in an attempt to make explicit 
links between the organizational structure of a school and its technical core, charter schools may 
advance new technical conceptions of efficiency not linked to institutionally accepted definitions, 
and the rejection of institutionalized definitions of efficiency may risk introducing a level of 
ambiguity that may prove too challenging to sustain. 


Organizational Structure. Recall that the normatively accepted and popular conception of 
effective schools is that they operate as rational systems that tightly coordinate the production 
of teaching and learning. The reality, however, is that schools operate as very loosely coupled 
organizations that are responsive to their institutional environment (Weick, 1976) and thus build 
gaps between their formal structures and technical core (J. W. Meyer et al., 1983). The formal 
structure is responsive to the environmentally created institutionalized rules and norms that 
define legitimate schooling, and it buffers teaching and learning activities (i.e., the technical core) 
from the demands of the institutional environment. However, when a school rejects structural 
conformity and instead chooses to organize around its technical core, it becomes subject to 
unbuffered scrutiny that may challenge the organization’s ability to document its efficiency and 
establish its legitimacy. The direct control and coordination of instructional activities and outputs 
“can uncover inconsistencies and inefficiencies and create more uncertainty than mere abstract and 
unenforced demands for conformity to bureaucratic rules” (J. W. Meyer & Rowan, 1978, p. 80). 


Accountability and Measuring Effectiveness. As charter school organizations face in- 
ward and become more attuned to their technical environment, then, according to the market 
forces theory of action, this new sector of decentralized schools will lead to the creation of a 
competitive education market. The education market will be driven by the quality of a school’s 
programmatic offerings defined by easily accessible and rationally organized academic per- 
formance indexes (in the form of output measures). Institutional theory suggests that “market 
mechanisms are presumed to be of primary importance in the case of technical environments. This 
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suggests that organizational conformity (whether structure, process, or outputs) is not compelled 
by formal rules or the threat of delegitimation, but by a concern for profitability and survival in the 
competitive market” (J. W. Meyer et al., 1983, pp. 63-64). In other words, schools that are more 
attuned to their technical environment will earn legitimacy in accordance to their performance 
in the local market structure and their ability to adapt to the market driven pressures that define 
effective education practices. 

The market-driven environment, however, will involve higher scrutiny of the technical core of 
an organization. As J. W. Meyer and Rowan (1977) explained, “when output can be easily eval- 
uated, a market often develops, and consumers gain considerable right of inspection and control. 
In this context, efficiency often determines success” (p. 354). Thus, the success or effectiveness 
of a school that is responsive to its technical environment is no longer determined by its ability 
to conform with institutionalized rules but rather by its ability to tightly coordinate and explic- 
itly link the means of production with measurable outputs. If charter schools cannot sustain a 
competitive advantage over traditional schools while attempting to advance a technical organiza- 
tional structure, their innovations may appear ineffectual compared to traditional institutionalized 
school structures and practices. 

The market forces theory of action, in particular, assumes that charter schools will operate 
in pure exchange markets, which will lead to greater organizational variability and improved 
school performance as schools organize in response to the technical demands of teaching and 
learning. But charter schools do not operate in perfectly competitive exchange markets, rather, 
they operate in “quasi-markets” that hinder the capacity for innovation and variation because of 
continued institutional pressures (Lubienski, 2003). The combination of quasi-market “failures” 
(Lubienski, 2003, p. 421) and continued pressures toward institutional isomorphism may ensure 
that charter schools will never operate in purely technical environments as originally hoped for 
by charter school proponents. 

In a sense, charter schools are caught in the middle between the market forces and local 
demands of technical environments on one hand and the isomorphic pressures of institutional 
environments on the other. In fact, absent a pure exchange market, charter school policies of school 
choice, competition, and regulatory autonomy may actually encourage isomorphism rather than 
work against it, as charter schools seek to gain legitimacy in what still amounts to be a very 
robust institutional environment of public education (Huerta, 2007; Lubienski, 2003, 2005). 
Recent research on charter schools supports this hypothesis, revealing greater convergence around 
organizational behavior than expected (Henig et al., 2005), as well as instructional and curricular 
conformity across the charter school population (Lubienski, 2004). 

Applying constructs from institutional theory as an analytical lens to the evolving theories of 
action will assist researchers in explaining and predicting the challenges that different charter 
school types may encounter as they evolve in the wider institutional environment of schools. For 
example, local control schools that promote idiosyncratic schooling models linked to specific 
community needs may result in a myriad of diffuse and differentiated schooling models that 
are challenged in their attempts to earn legitimacy and prove their efficacy outside traditional 
institutional definitions of effectiveness. Market-based schools, driven by their attempt to capture 
4 market share of students and realize profits, may integrate market-based efficiencies common 
in the private sector that may prove too unorthodox to the robust institutionalized definitions of 
schooling. In time, these unconventional practices may force schools to succumb to the isomorphic 
pressures that promote institutionalized forms of schooling. Finally, management recentralization 
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schools attempt to create economies of scale by mimicking traditional centralized structures that 
supply services and support to a wider network of schools aligned with explicit technical forms 
of schooling (without the goal of realizing a profit). By shifting the formal structure of schools to 
centralized network offices whose structures are linked to institutionalized sources of legitimacy, 
the network buffers school level actors and allows them to engage in more technical forms of 
schooling that are tightly linked to the internal goals of the CMO network. 


Microinstitutionalism 


The role of individual actors in organizations and their agency in navigating macro institutional 
forces are also important to consider in an analysis of evolving charter school organizations. 
Specifically, the analysis of institutionalized organizations must go beyond focusing on gover- 
nance and regulatory demands and should expand to include the actions of individual actors and 
their work in a macroinstitutional context, as they navigate the shifting policy, political, and legal 
contexts that are attempting to influence changes in schools today. This is consistent with the 
notion of an expanded definition of institutional governance that may be leading to more technical 
forms of schooling, beyond the traditional links to myths, ceremony and loose coupling (H. D. 
Meyer & Rowan, 2006a; Rowan, 2006). 

Micro-organizational analysis complements macro-organizational perspectives and provides 
insight into the nuances of how organizations develop and the agency that actors within organiza- 
tions possess in navigating institutional pressures—described by Zucker (1991) as the “cognitive 
processes involved in the creation and transmission of institutions” (p. 104). In our framework, 
we posit that accounting for the influence of microlevel actor agency will allow us to better 
understand how institutionalized scripts, which lead new school organizations to succumb to the 
isomorphic forces of the wider institutional environment, are interpreted and adapted by local 
actors (DiMaggio & Powell, 1983; J. W. Meyer & Rowan, 1977, 1978). 

Microinstitutionalism has important implications for analyzing and understanding the “dy- 
namic aspects of institutions” (Schneiberg & Clemens, 2006, p. 217) and understanding the work 
of evolving charter organizations. Through the microperspective, institutionalization is seen as 
a process in which individual or groups of actors champion new or alternative organizational 
models within an organizational field (DiMaggio, 1988; Schneiberg & Clemens, 2006; Tolbert 
& Zucker, 1994). In charter school reform, we suggest that the “empty institutional structures” 
(Kolderie, 2004) of charter schools create the space for different microlevel actors to mobilize 
around new and different models or logics of schooling. Although many models will tend to 
reproduce the macro order through processes of isomorphism, some invariably will not. But to 
carry new logics and organizational forms generated within individual charter schools outwards, 
“institutional entrepreneurs” (DiMaggio, 1988) working within the charter movement must build 
new institutional forms as a contingency to the challenges to scale. Charter school actors are 
in the process of building schools that reflect the diverse theories of action contained within 
the decentralized reform and their presence in the education field is consistent with a growing 
recognition in institutional analysis more broadly that organizational fields encompass multiple 
and competing logics, models, and organizational forms (Schneiberg & Clemens, 2006; Scott, 
2003). Thus, by exercising their agency within institutional environments of education, individual 
actors may build new organizations that support the transmission of culture across an increasing 
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number of schools and students, representing a viable and growing alternative to the existing 
institutional order of public education. 


CHALLENGES TO SCALE IN EDUCATION 


Macro- and microlevel institutional processes can be contextualized through an analysis of the 
challenges of scaling-up charter school reform: specifically, the technical, regulative, economic, 
political, and cognitive challenges that actors within school organizations confront in the process 
of building charter schools. 

Achieving scale within charter school reform has been a difficult proposition. Reformers have 
adhered to an “organic” logic of growth, by which new and innovative charter school ideas 
will somehow travel on their own volition from one school to another, absent a strategic and 
technical plan for institutionalizing new models of schooling across the wider educational field 
(National Charter School Research Project, 2007, p. 12). For example, prior to management 
recentralization, the ideas and practices of what some have called “break-the-mold” schools, 
such as Amistad Academy in New Haven, Connecticut, or KIPP Academy in the South Bronx, 
New York, remained isolated pockets of schooling, independent of a broader social context 
(Thernstrom & Thernstrom, 2003, p. 43). The assumption was that new ideas such as KIPP’s 
longer school day or Amistad’s use of interim assessments, would simply take hold and spread. 
The practice of management recentralization by CMO-operated charters, however, reflects a 
fundamental belief on the part of charter school leaders that large-scale change in education will 
not occur unless the charter movement creates institutional structures that facilitate the deliberate 
re-creation of successful school models. 

Building and sustaining new charter schools is in fact hard, demanding work that has created 
technical challenges for teachers and leaders, not unlike that of past generations of “break-the- 
mold” reformers described by David Tyack and William Tobin (1994). Wohlstetter and Griffin 
(2001), for example, spoke to the challenges inherent in the local control theory of action 
and described the tensions and overwhelming demands faced by charter school leaders when 
balancing instructional and managerial responsibilities in decentralized autonomous schools. 
Huerta (2009) described the economic and political challenges by arguing that public authority is 
not a commodity that can simply be devolved to local actors and result in empowerment. Instead, 
empowerment through decentralization is a process that must account for local economic and 
political conditions and the ability of local actors to activate public authority. 

Regulative norms may also challenge charter school expansion. Despite their freedom from 
the bureaucratic rules and regulations of the “one best system,” charters have for the most part 
tended to display little innovation and variation in organization and instruction (Lubienski, 2004), 
suggesting the influence of an institutionalized “cultural construction of schooling” that has been 
harder to alter than perhaps originally understood (Tyack & Tobin, 1994). According to this 
perspective, charter schools must look and act the part of “Jegitimate” schools (i.e., certified 
teachers, standardized curricula) to acquire the necessary resources for survival and growth, so 
innovation and variation is limited (DiMaggio & Powell, 1983; J. W. Meyer & Rowan, 1977, 
1978). 

Last, the declining rate of charter school openings indicates the supply of educators willing 
to open new charter schools appears nearly tapped out (Lake, 2007), suggesting a failure in 
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the institutional makeup of charter school reform to leverage different incentive structures to 
build a large-scale reform effort. Elmore (1996) emphasized how reform efforts often rely on 
the practice of a small group of intrinsically motivated teachers who are willing to learn new 
ways of organizing schools and engage in the arduous process of improvement. Absent a broader 
institutional reform agenda, charter schools have been no different than other reform efforts in this 
regard, concentrating small numbers of like-minded “believers” in isolated school settings where 
they have been free to experiment with different forms of organizational models and instruction 
independent of a broader social context. From Elmore’s perspective, such a structure may be 
problematic because it provides little incentive for all but a small proportion of the total teaching 
force to change—the ones who are highly motivated and excited to engage in serious reform in 
the first place. 

The challenges of scale in charter school reform, then, are similar to those of previous reform 
efforts—they are deeply rooted in the cultural norms of education institutions and in the failure 
of these institutions to change their incentive structures to align with broad-scale change. Unless 
modern reform efforts, like charter schools, seek to alter existing institutional arrangements and 
their associated incentive structures in clear and systematic ways, then they are destined to fail 
on a broad scale like many previous well-intentioned reform efforts (Elmore, 1996). Indeed, it is 
evident that scale may entail an acknowledgment of institutional barriers if schools are to address 
the more technical aspects of their practice in the process of redefining and transforming school 
organizations. Charter schools that attempt to promote more technical forms of schooling linked 
to the decentralized theory of action advanced by school operators will be forced to reconcile the 
demands of their local organizational goals with those of the wider and more robust institutional 
environment as they attempt to go to scale. 


CONCLUSION 


Charter schools are showing that innovations in teaching and learning and school organizational 
models are possible as a result of their decentralized context (Fuller, 2000; Huerta, 2000; Wexler 
& Huerta, 2000). However, the process of sustaining innovations is challenged by the institu- 
tionalized conditions that protect the status quo. Even incremental changes that could influence 
institutional shifts and accommodate innovations prove to be challenging under these conditions. 
Although it is clear that charter school reform has yet to yield large numbers of new, different, 
and better public schools, what remains less clear is how those that are succeeding in “breaking 
the mold” have done so, and how they are building new institutions that may in turn pressure 
change in the environment. 

Future research is needed to investigate both the micro- and macroenvironmental conditions 
that trigger isomorphism to understand how charter school organizations may counter the forces 
of isomorphism. New research must also begin to look at how charter schools are reacting to 
changes in their institutional environment, including the wider charter school movement, the 
changing discourse of regulatory schemes at the state level, and the local conditions that either 
foster or reject the development of new organizational forms. Research must also focus on gaining 
school-level insights that go beyond a snapshot description of charter school organizations and 
provide a longitudinal lens into the organizational evolution of decentralized charter schools as 
they navigate the institutional environment. 


INSTITUTIONAL THEORY ANALYSIS OF CHARTER SCHOOLS 429 


Further research of charter schools through the lens of new institutional theory will allow us to 
understand which decentralized theory of action may prove to hold the most promise in advancing 
new definitions of schooling that can challenge the institutionalized “rationalized myths” that 
define school effectiveness (J. W. Meyer & Rowan, 1977). The challenge for research is to 
understand the process of how charter schools evolve within the wider institutional environment 
of public schooling, and how the actors within charters reconcile macro-institutional forces with 
microlevel institution-building efforts that may lead to new definitions of effective schooling. 
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Educational privatization is rapidly expanding in many urban school districts, altering the social, 
political, and economic dynamics of educational policy and leadership. Yet many adherents cast 
privatization primarily as a fiscal or economic alternative to traditional public school management, 
ignoring these broader alterations. Drawing from a review of the educational privatization literature, 
as well as an analysis of current privatization reforms, this article offers an original typology of 
educational privatization and applies the typology to the reforms underway in New York City. 
It concludes with a discussion of the implications of this typology and privatization reforms for 
educational leadership practice and policy. 


Educational privatization is rapidly expanding in many urban school districts, with unprece- 
dented shifts in school leadership, governance, management, and accountability underway. In 
Philadelphia, Los Angeles, New Orleans, and Washington, DC, for example, private sector actors 
are partnering and contracting with school districts to provide supplementary services, manage 
entire schools, take over district-level administration, or provide access to private schools for 
voucher recipients. These efforts increasingly enjoy the support of policy elites in government 
and foundations as well as the tacit approval of many local school stakeholders, including new 
educational leaders, teachers, and community groups. Despite this growing support, however, 
privatization remains a controversial and contested issue in public education that significant 
constituencies, such as teachers unions, civil rights groups, and many community-based organi- 
zations, Oppose. 

Advocates of educational privatization often frame privatization as a last-ditch effort to re- 
dress educational inequality, especially racial achievement gaps on standardized assessments. 
They argue that with privatization come innovation, parental choice, efficiency, effectiveness, and 
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accountability. Meanwhile, critics oppose the prospect of profits being generated from underre- 
sourced schools and critique what they see as the loss of democratic governance and community 
input when privatization initiatives are enacted. Moreover, they question the qualifications of 
those who come into leadership and policy roles under privatization reforms. 

Amidst the debates, there is also disagreement about what “privatization” is. In the absence 
of a consensus, teachers, leaders, parents, and community members are often making decisions 
about whether and/or to what extent they will participate in reforms falling under the privatization 
umbrella with inadequate information about the scope and implications of their participation. 

Popular depictions of privatization often portray it as primarily a shift in fiscal management 
or economic processes, and ignore the social and political shifts it harkens (Bloomberg, 2007). 
In addition, much of the scholarship and debate on privatization positions it as a reform that is 
diametrically opposed to public sector management when increasingly private sector and public 
sector actors and institutions are intertwined. These renderings fail to provide a more complex 
exploration of privatization that can clarify what it is, how it is expanding, where it is taking hold, 
what implications it has for urban school reform, and why particular school districts seem to be 
hotbeds for privatization activity. 

In contrast, Martha Minow (2002, 2003) argued that the private sector need not be seen just 
as rivals but also as partners to the public sector. Her conceptualization of privatization calls for 
a broader public debate about the possible contributions the private sector can make as partners 
to government in providing social services, rather than merely operating as rivals. This debate 
would include deliberation about the political and philosophical aspects of such partnerships in 
terms of democracy, accountability, transparency, and equity. 

This article examines the proliferation of systemic educational privatization! and offers an 
original typology that conceptualizes the institutional arrangements that emerge under privati- 
zation reforms. We build upon Minow’s “partners or rivals” frame, drawing from a review of 
empirical research as well as from political and philosophical analyses of educational priva- 
tization. Our typology is also informed by our previous individual and collaborative work on 
educational privatization in New York City (DiMartino, 2009a; Scott, 2008; Scott & DiMartino, 
2009; Scott & Fruchter, in press). 

We argue that prevailing definitions of educational privatization tend to underemphasize its 
political and social aspects, and instead focus on the administrative, economic, or technical 
characteristics (Wells & Scott, 2001). The economic aspect refers to a shift from public to 
private provision and/or financing of educational services (Donahue, 1989; Moore, 2003), but 
privatization processes also change the ways schools are managed, financed, governed, and held 
accountable. We define privatization as a range of reforms that redistribute resources and control 
over most aspects of schooling away from traditional public governance structures to a disparate 
assemblage of parents, teachers, school leaders, community members, private sector actors, and 
private organizations. 

Our analysis is organized into four major sections. First, we briefly examine the social history 
and development of privatization in the United States. Second, based on the themes generated in 
this review, we propose a new typology of educational privatization that keeps pace with recent 


| We define systemic privatization as the comprehensive, private sector-reliant activities in school districts that involve 
not only multiple district operations but also those within local schools (Feigenbaum & Henig, 1994). 
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urban school privatization. Third, we apply the typology to, the privatization reforms underway in 
the New York City Department of Education (NYC DOE). Finally, we discuss the implications 
of this typology and New York City reforms for educational leadership. 


EDUCATIONAL PRIVATIZATION: IDEOLOGY, RACE, AND SCALE 


Competing ideologies regarding the role of government, civil society, and the private sector in 
the financing and provision of social services animate educational privatization debates. Despite 
these debates, a seeming policy consensus about the merits of privatization for urban school 
district improvement—at least judging by the similarity in privatization reforms being adopted 
across urban systems—appears to exist among decision makers. 

As many researchers have observed, the use of the private sector in education has a long 
history. It has traditionally involved pragmatic strategies for cost saving or efficiency—such as 
the common practice of districts contracting out for particular services (Carpenter & Hall, 1971; 
Hannaway, 1999; Hill, Pierce, & Guthrie, 1997; Kemerer, 2000; Lin & Hassel, 2001). 

Within the last three decades, however, educational privatization has been attached to an 
ideological argument about the superiority of the private sector over the public, especially with 
regards to remedying what many conservative and neoliberal advocates regard as the failure of 
public schools to provide excellent education given public resource allocations (Henig, 1990). 
These advocates, influenced by the work of Milton Friedman (1962), regard public education 
bureaucracies as monopolies beholden to the constraints of unions with little market incentive to 
fundamentally change practices. 

More recent realizations of neoliberal and conservative visions for schooling include a retreat 
from race-conscious reforms of recent decades and a radically altered state role in the provision 
of public schooling, with private sector actors at the leadership helm, and with equity redefined as 
parents’ unfettered ability to choose schools for their children. Recently, some conservatives and 
progressives, including new civil rights organizations, have found philosophical common ground 
in the potential of privatization and choice to provide more equitable schooling by empowering 
parents with the choice to flee struggling public schools (Scott, Lubienski, & DeBray-Pelot, 
2008). Still, teachers unions and community and parental advocacy groups have also fiercely 
contested privatization and market-based choice reforms (Cortez, 2008; Scott & Fruchter, in 
press). 

Despite the recent federal government bailout of the financial sector, which some economists 
argue has resulted in the de facto nationalization of the banking industry (Cho, Irwin, & Whoriskey, 
2008), the United States has not historically maintained a stable of nationally owned enterprises 
to be privatized, as has been the case in other industrialized countries. In the United States, privati- 
zation of social services has been more piecemeal; officials have episodically contracted with the 
private sector for military services, printing, trash collection, and food services, for example. Con- 
tracting accelerated in the late 20th century, largely because of the taxpayer revolts of the 1970s 
and early 1980s (Handler, 1996). Taxpayers became frustrated with the increased government 
spending on social welfare programs, and persistent inflation left the middle and working class 
worried about their futures. Much of this angst was a White and middle-class backlash against civil 
rights redistributive social policy; these groups blamed the country’s gloomy economic outlook on 
the government’s efforts to remedy past injustices (Edsall & Edsall, 1992). Calls for a reduction in 
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government, opposition to school bussing to achieve desegregation, and the expansion of the sub- 
urbs all energized restructuring efforts that would pave the way for current iterations of educational 
privatization. 

These antiwelfare state sentiments led to a series of deregulatory policies across national, state, 
and local governments. With the removal of governmental oversight, contracting for services 
increased in several public sectors, public education among them. In the early 1970s, more than 
150 school districts contracted with private companies for instruction in various subjects. The 
intent was to increase the academic performance of poor and minority students, but the results were 
decidedly mixed and revealed the difficulty contracting can pose in terms of costs, transparency, 
competitive bidding, oversight, and holding contractors accountable for results (Ascher, Fruchter, 
& Berne, 1996; Carpenter & Hall, 1971; Gramlich & Koshel, 1975; Handler, 1996). 


Deregulation, Decentralization, and Market-Based Reforms 


The Reagan administration-commissioned report, A Nation at Risk (National Commission on 
Excellence in Education, 1983), lamented the nation’s schools drowning in a “sea of mediocrity” 
(p. 1). In its aftermath came a series of state-level reforms aimed at increasing academic standards 
and implementing more stringent accountability measures. Influenced by corporate models, dis- 
tricts began experimenting with decentralization. Throughout the 1980s and 1990s, site-based 
management, a popular form of professionally oriented decentralization, allowed principals and 
school building leaders more control over budgets, personnel, and curriculum. The assumption 
was that increased local autonomy would produce better achievement for students. At the same 
time, efforts to enact school vouchers, never dormant, were successful in Cleveland, Ohio, and 
Milwaukee, Wisconsin. Charter school legislation passed, beginning in Minnesota in 1991 and 
spread to 40 states by 2009. Although site-based management, vouchers, and charter schools are 
informed by neoliberal and conservative ideologies, there has also been consistent liberal and 
progressive advocacy for these measures as well (Apple & Oliver, 1996; Pedroni, 2007; Wells, 
Lopez, Scott, & Jellison-Holme, 1999). Moreover, many of the original community control ac- 
tivists of the 1960s and 1970s have found a renewed space for advocacy in the voucher and charter 
school movements. 

Across the United States, many state-level and urban school system leaders are currently 
adopting systemic privatization, in which public funds are redirected to organizations whose 
commitments to broader social goals of equity and democracy are unclear or unstated. Ma- 
jor developments include the proliferation of for-profit educational management organizations 
(EMOs), nonprofit charter school management organizations (CMOs), intermediary organiza- 
tions, and partnership organizations in the operation of schools, largely facilitated by school 
officials. Notwithstanding intriguing partnerships that have the potential to increase the capacity 
of schools and districts, these organizational models raise significant tensions regarding the abil- 
ity of parents, teachers, and school leaders to influence the curriculum, governance, and other 
policies of local schools. 

In the aftermath of 2005’s Hurricane Katrina, New Orleans schools have been almost com- 
pletely “charterized;” most of its schools are run by EMOs or CMOs. Philadelphia’s school 
system underwent state takeover in 2002, and its School Reform Commission recommended that 
the district adopt a diverse provider model wherein EMOs, CMOs, and other nonprofits would 
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partner with or manage schools (Bulkley, Mundell, & Riffer, 2004; see also Useem, 2009/this 
issue; DiMartino, 2009a). Similar shifts in the management of schools are underway in Los 
Angeles and New York City, largely funded by foundations interested in increasing the market 
share of management organizations in particular urban school districts (Scott, 2009). 

Some researchers have found that these new arrangements have resulted in higher achieving 
schools and greater efficiencies, but these data are highly contested (see, €.g., Lubienski, Weitzel, 
& Lubienski, 2009; Peterson, 2007). Although there are potentially more opportunities for school 
leaders to enjoy autonomy as building and system “CEOs,” privatized schools also present policy 
tensions in terms of greater student selectivity, diminished public participation and democratic 
accountability (Woodworth, David, Guha, Wang, & Lopez-Torkos, 2008). Political examinations 
of privatization have considered power relationships between various educational stakeholders, 
because participants in privatization reforms potentially have access to unequal influence. This 
research has explored how teachers, school leaders, and private sector actors negotiate decision 
making while making sense of new institutional arrangements, governance mechanisms, and 
accountability systems, especially under privatized school management (Archer, 2004; Bulkley 
et al., 2004; Conn, 2002; Miron & Nelson, 2002; Welles, 2000; Whittle, 2005). 

Margonis and Parker (1995) argued that choice and privatization measures in urban districts 
represent strategies of “containment;” they are attempts to keep poor children of color enclosed 
within urban districts and out of more wealthy suburban settings. Urban school district privati- 
zation is often state-initiated and based on arguments that school failure is solely the result of 
local district leadership. This allows for managers, who are more likely to be White and male 
and to have less experience in public education than existing school district leaders, to emerge 
as the new leaders of segregated schools and school districts (Maranto, 2005; Rhim, 2007; Scott, 
2008). The school management arena, then, becomes a key area of privatization where alterations 
in educational leadership are especially prominent and politically charged. Other privatization 
forms intersect and expand this shift in leadership; for example, alternative certification programs, 
which are designed to attract leaders from outside traditional public school leadership networks 
and largely funded by foundations. Two such programs in New York City are New Leaders for 
New Schools and the New York City Leadership Academy. 

Given the diversity, growth, and overlap of privatization forms in the educational privatization 
movement, an updated framework is needed to describe and interrogate it and to help aspiring 
and current urban educational leaders understand the new organizational environments in which 
they will likely be working. An exploration of the types of privatization can also inform a more 
nuanced public debate about which forms are desirable and why, and which fail to serve the 
interests of students of color, as well as the broader public purposes of public education. 


A NEW EDUCATIONAL PRIVATIZATION TYPOLOGY 


Our survey of existing typologies determined that many were developed before the most recent 
expansion of privatization initiatives, and were in need of updating (Murphy, 1996). Several 
typologies come from research on the British implementation of market-based and privatization 
reforms. Although they are instructive about systemic privatization in a highly marketized policy 
environment, they also have limited applicability to the U.S. social and political context (Gewirtz, 
2002; Whitty & Power, 2000). 


Author 


Feigenbaum & Henig 
(1994) 

Woods & Woods 
(2005) 


Murphy (1996) 


Minow (2002) 


Wells & Scott (2001) 


Gilmer (1997) 


Henig, Holyoke, 
Lacireno-Parquet & 
Moser (2003) 


Perry & Rainey 
(1988) 

Smith & Wohlstetter 
(2006) 


A Survey of Existing Privatization Typologies 


Sector 


K-12 Education, 
Public Sector 
Primary Education, 

England 


K-12 Education 


Public Services: 
Education, Welfare, 
Medicine, and Law 

Charter Schools 


Higher Education 


Charter Schools 


Public and Private 
Organizations 
Public and Private 
Partnerships 
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TABLE 1 


Conceptual Focus 


Political goals and 
consequences 

Roles of private sector in 
public education 


Administrative and 
governance shifts under 
privatization 


Political deliberation; 
Public values and social 
consequences 

Sociological analysis with 
equity focus 

Fiscal and Economic 


Critique of the dualist 
theory of public and 
private sector 


Organizational 
characteristics 

Organizational 
differentiation 
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Dimensions 
Pragmatic, Tactical, and Systemic 


Cultural Transformer, Expert 
Provider, Therapist, Partner, 
Network Colleague, Catalyst 

Load Shedding, Asset Sales, 
Volunteerism, Self-Help, User 
Fees, Contracting, Franchises, 
Vouchers, Grants/Subsidies, 
Deregulation 

Partners and Rivals 


Political, Social, and Economic 


Public Production with Public 
Finance, Public Production 
with Public Finance in Higher 
Education, Private Production 
with Government Financing, 
and Private Production 

Government, Market, Voluntary 
Nonprofit; across four 
dimensions: Finance, Delivery, 
Responsibility, and Equity 

Ownership, Funding, Mode of 
Social Control 

Process of partnership initiation, 
types of services provided, 
partnership form, and depth of 
organizational involvement 





Among the existing privatization typologies, perhaps the most salient to contemporary U.S. pri- 


vatization comes from Feigenbaum and Henig (1994). Their typology employs three categories: 
pragmatic, tactical, and systemic. Pragmatic approaches mirror the administrative conceptual- 
ization, but “the political perspective considers such pragmatic privatizations to be discrete and 
context-dependent episodes” (p. 191). Tactical privatization is meant to enhance the position 
of the privateer, which can include political parties, politicians, or interest groups. The final 
approach is systemic privatization. This is the most comprehensive form and 1s intended to re- 
structure political and economic institutions. It has three aims: lower the public expectations for 
what government should provide, diminish governmental oversight and enforcement, and alter 
interest groups’ support for government growth (p. 192). These researchers argue that systemic 
privatization shifts not only the capacity of stakeholders to pursue agendas but also the values, 
culture, and expectations of the public and the institutional arrangements of society to encourage 
greater reliance on the private sector. Table | summarizes other existing educational privatization 
typologies, noting their conceptual orientations, privatization dimensions, and schooling sector. 
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FIGURE 1 Privatization typology. 


Although these typologies are useful, many were developed before the creation and expansion 
of charter school reform, the development of management organizations, the passage of No Child 
Left Behind (NCLB), the proliferation of venture philanthropies whose funding agendas support 
privatization, and the expansion of the testing and data industries. As a result, they fail to capture 
the range of systemic privatization actors in many urban districts. 

Based on our review of the conceptual, ideological, and empirical literature on privatization, 
as well as a review of existing privatization typologies, we constructed a new, comprehensive 
typology of educational privatization. This typology attends to the practices, structures, individual 
actors, and processes that comprise privatization reforms in urban school districts. It begins where 
Feigenbaum and Henig’s ends: describing the types of privatization actors in a systemically 
privatized environment, Building on Minow’s (2002) notion of partners and rivals, our typology 
of educational privatization expands this framing to encompass five forms: (a) Gatekeepers, (b) 
Partners, (c) Rivals, (d) Managers, and (e) Profit Seekers. For the purpose of explicating each one, 
we provide a distinct discussion of these five types, but we note that there is important overlap 
among the five and that in many urban districts like New York City, systemic privatization is 
underway with all five types operating in different degrees across the school system. Figure 1 
shows this typology and provides examples of each type. 


Gatekeepers 


This type of privatization refers to individuals and organizations that facilitate privatization policy, 
including policymakers, district officials, and courts, and to the practices that local schools 
engage in to limit enrollment, raise funds, and “brand” themselves. Examples of individual 
gatekeepers include state policymakers, whose legislation in Louisiana and Pennsylvania enabled 
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the New Orleans, Philadelphia, and Chester school systems to turn over many of their schools to 
management organizations. At the district level, superintendents and school boards often provide 
private sector actors with the opportunity to provide services or school management by issuing 
requests for proposals for specific school and district contracts. This is especially common when 
the superintendent or district leader comes from the private sector, as is the case in New York 
City (Wells, Slayton, & Scott, 2002). Gatekeeping philanthropic foundations have also facilitated 
the growth of privatization, especially that of EMOs and CMOs. At the local level, small schools, 
charter schools, and other marketized schools demonstrate gatekeeping practice when allowed to 
set admissions criteria. Even though they typically do not explicitly violate federal civil rights 
statutes, often these selective practices contribute to both the “creaming” and “cropping” of 
student enrollments (Lacireno-Paquet, Holyoke, Moser, & Henig, 2002). The gatekeeping form 
of privatization, then, describes individuals, organizations, and the practices they initiate that 
create the environment for private sector actors to operate in a public education environment. 


Partners 


Partners are individuals and organizations who enter into collaborative relationships with schools 
and school districts, often contributing as much as they gain from the relationship. Universities, 
for example, often partner with local schools; they offer faculty expertise while schools offer 
opportunities for college students to do research or apprentice as student teachers or leadership 
interns. Other examples of partners are school reform consortia (Coburn, Bae, & Turner, 2008), 
community-based organizations, business groups, and media outlets. Cultural and civic groups 
also frequently partner with local schools or school districts. Of course, these partnerships are not 
exempt from the policy tensions we have discussed in terms of the different levels of power enjoyed 
by schools, districts, and institutional partners. Some partnerships, rather than functioning as 
relatively equal interactions between institutions, in fact look more like management relationships, 
especially when the partner has needed resources or expertise that the local school or district lacks 
(DiMartino, 2009a; Honig, 2009/this issue). 


Rivals 


Rivals are private sector actors whose energies or organizations are geared toward competing 
with traditional public schools—for students, results, and/or resources. These players are often 
informed by market ideology; they believe that school improvement will only happen when public 
schools are forced to compete, and that opening up the schooling sector to the private sector can 
help to foster risk taking and innovation. Voucher advocates and plans, and some charter school 
founders and networks are key examples of rivals to public schools. Ultimately, the goal of rivals 
is to expand their market share and put low-performing public schools “out of business.” 


Managers 


The fourth type of privatization are managers, and as we have argued, this is the fastest grow- 
ing type of privatization actor, especially in urban school districts. These are individuals or 
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organizations oriented toward whole school or partial school district governance. They aspire to 
make key decisions over schooling and seek the authority and public funding they deem necessary 
to execute their work. EMOs, CMOs, intermediaries, and partner support organizations fall under 
this category. The growth of this form of privatization has been assisted by the adoption of charter 
school policies across 40 states, which has encouraged the growth of the school management 
sector (Miron & Nelson, 2002). As the charter school movement has matured, autonomous, 
locally grown schools are less common. More prevalent now are charter school networks and 
management organizations that manage franchises of schools across different districts and states. 
Philanthropists have dedicated millions of dollars to encourage the growth of such management 
organizations that boast high student achievement in their school in urban districts. Due in part 
to foundations’ financial largesse, the majority of charter schools in many cities are operated by 
management organizations (National Charter School Research Project, 2007; Scott & DiMartino, 
2009). Attempting to build an identifiable brand and to demonstrate academic results to policy- 
makers and donors, these organizations often insist that local schools adhere with fidelity to their 
particular school model, its curriculum, governance structure, admissions criteria, and personnel 
policies. Although uniform practices are common in corporate settings, in public school settings, 
tensions can emerge when brand homogeneity conflicts with local desires to shape the schooling 
environments according to the needs of particular students or communities, or to reflect particular 
racial, cultural, or linguistic identities. 


Profit Seekers 


Finally, we term the last type of privatization actor profit seekers. Here we refer to organizations or 
individuals whose central motivation is to make money. Although this does not preclude them from 
having multiple motivations—student achievement, parent satisfaction, school improvement, 
public relations, and growing their brand, for example—they are unlikely to engage in any of 
these secondary goals if doing so will diminish their ability to make money. Textbook and test 
preparation companies are key examples of this category, as are some management organizations, 
data analysis firms, food service providers, and financial service organizations. 

We reiterate that significant overlap exists across these types, and few individuals or organi- 
zations fall into just one. Although this typology is a work in progress, it captures the breadth 
and depth of private sector involvement in public education and allows for an examination of the 
systemic privatization many urban school systems are currently undergoing. In the next section, 
we apply the privatization actor types to an examination of the privatization reforms in the New 
York City public schools. 


PRIVATIZATION IN NEW YORK CITY 


New York City has experienced an influx of private providers with different orientations to 
schooling, profit, and equity. This enables us to apply our framework and demonstrate how the 
categories of private sector involvement that we have identified play out in practice. 
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Gatekeepers 


One function of gatekeepers is to decide which groups or individuals can enter an environment. 
Given their tremendous decision-making power, it is often the gatekeepers’ values and goals for 
schooling that dominate a system, or in the case of education, a district or a school. In 2002, 
when the state legislature gave the mayor control of the public schools, he became the ultimate 
gatekeeper in New York City. Upon gaining control of schools, Mayor Bloomberg, the former 
CEO and founder of Bloomberg LP, chose to hire corporate sector professionals to be key leaders 
within the DOE. For example, he hired Joel Klein, the chairman and CEO of Bertelsmann, Inc., 
to be chancellor of the New York City Public Schools. In turn, Chancellor Klein hired McKinsey 
and Company, and Alvarez & Marsal, private management consulting firms, to help with the 
reorganization of governance and operational structures within the NYC DOE (Gootman & 
Herszenhorn, 2007). Chancellor Klein hired Chris Cerf, the former president of the EMO, Edison 
Schools, Inc., to be the deputy chancellor of operational strategy, human capital, and external 
affairs. Espousing market ideologies and the positive potential of competition, these leaders 
invited private sector organizations to partner with the DOE to provide educational services to 
further their visions for school reform. 

At the mayor’s and chancellor’s bidding, private sector organizations have been involved in 
starting schools, reorganizing the DOE’s governance system, creating systemwide data man- 
agement programs, and contracting with advertising agencies to rebrand academic achievement. 
Funding for many of these new initiatives and programs comes from philanthropists and hedge 
fund managers rather than public coffers. This influx of private money into the DOE has facilitated 
the speedy adaptation of many new initiatives and programs. 

The creation of more than 200 new small schools in partnership with private sector organiza- 
tions is another Bloomberg gatekeeping practice. These new small schools often have connections 
to important networks, are clearly branded with the name of their private sector partner, and have 
individualized admissions policies. The College Board, famous for the SAT, is an example of one 
of these private sector organizations involved in school reform. Schools founded in partnership 
with the College Board receive not only additional funding but also in-kind services and resources, 
such as access to the College Board’s Advanced Placement curricula, testing services and even 
to graphic designers who help the new schools create their logos. In return for this support, 
these hybrid, public—private schools must use the College Board’s English, math and college 
preparation curricula in Grades 6 through 12 and adopt their professional development programs. 
Most visibly, however, these schools must include the phrase “College Board” in their name, 
thus creating school monikers such as Pathways College Preparatory: A College Board School. 
Although these schools do not admit students based on prior demonstrations of achievement, 
they do require their incoming sixth-grade students to submit applications and be interviewed 
by the school principal or lead teachers, thus adding to their admissions processes’ gatekeeping 
practices that are usually associated with private schooling. 

The opening of the new small schools, as well as charter schools, increases the schooling 
options available to students and their families while building an education marketplace in 
New York City. The creation of this marketplace represents an integral part of the chancellor’s 
Children First reform initiative that seeks to create a portfolio of schooling options from which 
students and their families can choose. In turn, schools are expected to compete for students. To 
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help families choose from among schools and to hold principals accountable for their schools’ 
outcomes, the DOE has created new ways to measure school effectiveness: progress reports 
and quality reviews. With the progress reports, each school receives a letter grade (A-F). These 
grades are based on standardized test scores, attendance reports, and the results of the schools 
surveys administered in the spring of 2007 (New York City DOE, 2007). In contrast to the 
outcome-oriented progress reports, the quality reviews provide a more descriptive overview of 
the school. Data are collected during 1 to 2 days of interviews and observations at the school. After 
the completion of the review, schools are given one of five possible scale scores ranging from 
“outstanding” to “underdeveloped.” In addition to a summative rating, schools receive written 
feedback about their performance (see Shipps & White, 2009/this issue). 

In tandem with the emphasis on outcomes has been a move by the DOE to increase students’ 
motivation to achieve academic success. To raise the scores of low-performing students, in par- 
ticular poor children and children of color, DOE officials enacted or will enact various incentives 
programs ranging from paying students for perfect test scores to rewarding high-performing 
students with free cell phone minutes and tickets to sporting events (Bosman, 2007; Medina, 
2007b). For example, a recent million-dollar donation by Whitney Tilson, a hedge fund manager 
and vocal school choice advocate, will pay students for receiving passing scores on Advanced 
Placement exams. In his program, students receive $1,000 for scoring a 5, $750 for a 4, and $500 
for a 3. Currently, this program exists in 25 public schools in New York City (Medina, 2007a). 
As this example illustrates, gatekeepers’ access to private funds, whether top education officials 
or private funders themselves, allows them to make educational policy without the democratic 
participation and transparency traditionally expected of decision-making processes in the public 
sector (“New York’s Schools,” 2007). 

In these initiatives, we see the intersection of the accountability domain with the gatekeeping 
type of privatization actor; Samsung is a sponsor of the cell phone program, for example. 
This program, part of the city’s “Million Motivation Campaign” is the brainchild of Harvard 
economist Roland Fryer, the DOE’s chief equality officer, and supervised by Bruce S. Gordon, 
a former Verizon executive and former president of the NAACP. The DOE, as part of their 
larger rebranding effort, can utilize the cell phones to send text messages to students about the 
importance of education. Students in this program who achieve academic success could also 
receive other rewards, most of which will be supplied by private sector partners (Medina, 2007b, 
2008a). 

When the program was implemented (2007-08) in seven selected Brooklyn schools, four of 
which were affiliated with the Knowledge is Power Program (KIPP) charter school network, 
many phones were awarded to students who were already excelling (Medina, 2007b, 2008a). An 
external evaluation conducted by Fryer will determine if external incentives increase academic 
motivation; it is unclear how the evaluation will account for the selection of already high-achieving 
students as cell phone recipients when determining the program’s academic value.” In the fall of 
2007, New York City launched another incentive program. This pilot program, which exists at 
40 schools, pays fourth and seventh graders $25.00 and $50.00, respectively, for earning perfect 
scores on each of the 10 standardized tests administered over the course of the school year 
(Medina, 2007c). Currently, foundations and philanthropists fund this program (New York City 
Office of the Mayor, 2007). 


2The DOE also banned cell phones in all city schools over significant parental opposition. 
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Borrowing performance incentives usually found in the corporate sector, education officials 
have increased their reliance on outcomes-based accountability mechanisms to evaluate students 
and schools. This increased reliance on outcomes, as measured by standardized test scores, means 
that inputs such as teacher quality and retention, and class size reduction receive less attention 
and funding. As this section illustrates, the ideologies and actions of gatekeepers directly affects 
what types of programs and initiatives receive funding, the type of expertise needed to run 
these new programs, how student and school success is measured, and the mechanisms used 
to motivate students. Although the individuals associated with gatekeeping influence decision- 
making processes within New York City’s public education system, other players, such as the 
district’s partners, rivals, managers and profit seekers, seek to use their outsider status to influence 
educational decision-making. 


Partners 


The use of intermediary organizations to found and run new small schools and the emergence 
of partner support organizations to provide technical support to public schools represent the 
“partners” aspect of educational privatization. 

The involvement of private sector partners, called intermediary organizations in New York 
City’s new small schools movement, is an example of both school-level “partners” and “managers” 
under our framework. Since 2002, more than 260 new small schools have opened in New York City 
(NYC DOE, 2008). The NYC DOE in partnership with an intermediary organization founded 
a majority of these schools. Intermediary organizations receive funding from foundations to 
cofound and run new small public secondary schools. Intermediary organizations include the 
aforementioned College Board, New Visions for Public Schools, and the City University of New 
York. For example, in 2005, the Bill and Melinda Gates Foundation (2005) gave $11,850,000 to 
the College Board to start new small schools. 

The introduction of Partner Support Organizations (PSOs) as part of the current restructur- 
ing of the NYC’s DOE provides another example of the Bloomberg administration’s push for a 
systemwide increase in private sector “partners” and also signals the potential entry of “profit 
seekers” into public school management because the request for proposals was open to both 
nonprofit and for-profit organizations. As of the 2007 reorganization, the NYC DOE is no longer 
divided into 10 distinct regions; now schools choose between three different types of school sup- 
port organizations for what were once regional support services. Two of them, the Empowerment 
School Organization and the four Learning Support Organizations, remain internal to the NYC 
DOE. However, the third grouping of school support organizations, partner support organizations 
(PSOs) hails from the private sector (NYC DOE, 2007). For-profit and nonprofit organizations, 
then, have an unprecedented opportunity to build their market share of school services like profes- 
sional development and special education while they support the restructuring effort. In addition 
to being partners, and potentially profit seekers, because the PSOs compete against the two inter- 
nal support structures, they also act as rivals to the public system. Rivals in other arenas include 
charter schools, EMOs, and for-profit curriculum and professional development providers. 


Rivals 


Charter schools and schools managed by EMOs or CMOs represent clear rivals to traditional 
public schools governance structures. Other, subtler rivals, such as the Princeton Review and 
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Kaplan K12 Learning compete in their domains of teaching, learning, and diagnostic test- 
ing. New York City’s leadership has embraced charter schools as much to expand choices 
for students in struggling schools as to encourage competition between schools. Currently 
60 charter schools operate in New York City with an additional 50 more expected to open 
in the next 3 to 5 years as a result of state legislation. raising the cap on charter schools. 
Eighteen of them were to open in September 2008, most of which would be managed by a 
CMO. Although some community-based or nonprofit organizations such as KIPP run charter 
schools, for-private educational management companies manage others. These management or- 
ganizations hold the sometimes competing goals of raising student achievement and gaining 
financially. Two examples include Victory Schools, Inc., and Edison Schools. At the same time 
that these new rival school managers have emerged, there has been an increased reliance on 
alternative certification programs, such as New Leaders for New Schools and the New York City 
Leadership Academy, to provide principals to staff the new schools. The alternative programs 
rival current university-led preparation and training and suggest the emergence of a new set of 
criteria for school leaders. 

The advent of NCLB and its increased reliance on testing indicators have brought new players 
into the arenas of curriculum and professional development. Increased federal pressure on schools 
to reach annual yearly progress targets, coupled with city pressure on teachers and students to 
past high-stakes tests, has pushed school leaders away from using teacher-, school-, or district- 
generated curriculum in favor of curricula that directly addresses test preparation. Currently 
Kaplan, famous for its standardized test preparation programs, offers elementary curricula in 
both English and math as part of their Kaplan K12 Learning Services. In 2003-04 one region, 
which enrolled 91,224 public school students across Brooklyn and Queens, hired Kaplan to 
provide instructional materials to students and professional development to teachers (Kaplan 
K12 Learning Services, n.d.). Another test preparation service, The Princeton Review, offers 
similar curricular and professional development programs. Purchasing curricular materials and 
professional development from private sector, for-profit vendors is not new, however, the increased 
reliance on measurable outcomes, such as test scores, has pushed school leaders to seek out 
standardized curricula and test preparation guides to prepare their students and teachers to function 
within this accountability driven environment. In this climate, test preparation organizations not 
only act as “rivals” to other school- or region-created curriculum developers but also as profit 
seekers from the public school coffers. The testing industries’ notably increased market share in 
education is discussed in the upcoming Profit Seekers section. 


Managers 


The creation of a “portfolio” of schooling options from which students and parents can choose 
remains a hallmark of Bloomberg’s and Klein’s Children First Initiative. Bloomberg and Klein 
pushed for the introduction of new “managers” of schools to build this portfolio. Examples of 
new managers in New York City include Edison Schools and Victory Schools, Inc., the Urban 
Assembly and New Visions for New Schools. Discussed in the Rivals section, EMOs such as 
Edison Schools and Victory Schools, Inc., represent one type of manager, in their cases for- 
profit EMOs that run schools in New York City. However, over the past 2 years, other types of 
managers have begun to oversee schools in New York City. In 2006, the Urban Assembly signed 
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a “prototype management contract” with New York City to manage 19 schools, which founder 
Richard Kahan, a former real estate developer, called “an unprecedented management agreement” 
(The Urban Assembly, n.d.). The contract gives Urban Assembly, a nonprofit organization, “a 
large role, including say in hiring principals and administrators” (Herszenhorn, 2006, p. 2). The 
Urban Assembly, which started as an intermediary organization by cofounding and running small 
schools, now acts as its own miniregion or district within New York City (The Urban Assembly, 
n.d.). 

Similarly, by opting to become a PSO within NYC DOE’s newly reorganized governance sys- 
tem, New Visions for Public Schools, as of July 2007, has its own contract to provide technical 
support in teaching and learning and data analysis to city schools. Other PSOs with separate 
technical support contracts include Replications, Inc.; The Academy for Educational Develop- 
ment; and Fordham University. The Urban Assembly contract and the PSO contracts represent a 
move by New York City’s educational leaders to privatize services formally managed by region 
or district personnel. 


Profit Seekers 


A $17 billion district budget creates the potential for extracting a profit from delivering educational 
services in New York City’s public schools. By providing beverage services, data management 
expertise, curriculum, and after-school tutoring, Snapple, IBM and Kaplan K12 Learning Services 
are a few examples of for-profit companies that have sold their services over the past few years to 
the DOE. In 2003, Mayor Bloomberg signed a contract with Snapple, making it the only beverage 
brand sold in the city’s public schools. In exchange for the brand exposure and profits, New York 
City received a 30% commission and $3 million donated annually to school sports programs. 
Although proponents argued that the Snapple deal brought extra revenue to public school system, 
detractors questioned the lack of transparency associated with the contract and opposed the use 
of schools for commercial means, especially the marketing of high-calorie drinks to children 
(Lucadamo, 2006). 

New York City’s recent 5-year contract with IBM to create a comprehensive data management 
program also garnered mixed reviews. DOE officials argue that the program, called ARIS, brings 
together student-level demographic and assessment data with school-level data gathered from 
school quality reviews, progress reports, and surveys to provide teachers and administrators with 
comprehensive information to allow them to more effectively track individual student and school 
progress. ARIS is not available to parents and students for confidentiality reasons. Critics argue 
that the money spent on the $80 million contract could have been used inside the DOE to build 
internal data analysis capacity, or to lower class size, buy more curricular materials and supplies, 
or build new schools (Einhorn & Saul, 2007). 

The accountability provisions in the federal NCLB have provided a fertile market for test 
developers, and test preparation companies to contract with districts and individual schools to 
sell their test preparation and data analysis programs. As mentioned, companies such as Kaplan 
and The Princeton Review have become key players offering an array of K-12 test preparation 
services. In addition, to providing test related school services during the school day, both of these 
organizations, as well as many others, have become Supplemental Educational Service providers. 
They offer tutoring and other enrichment activities to help students in failing schools improve their 
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achievement in reading, math and language arts. Supplemental Educational Service providers act 
both as “profit seekers,” making money from the services that they offer, but also as “partners” 
and “rivals” to the public school system. They are partners because public school students benefit 
from the extra services they provide, but they are rivals because they also provide a service that 
could otherwise be provided internally. As frequently illustrated in this description, overlap exists 
among the five types of privatization actors. It is this overlap and the consequent blurring of 
our typological boundaries that necessitates further interrogation of the multiple dimensions of 
educational privatization, beyond what is described here. 


DISCUSSION AND IMPLICATIONS 


Although it is too early to draw final conclusions about privatization restructuring in New York 
City, it is clear that the increasingly privatized leadership and management of the district and 
many of its schools has raised questions that require closer attention and public deliberation. 
These include the role of democratic governance, the state of educational leadership, and the 
implications for racial equity in public education. 

Although the DOE boasts about important benchmarks it has hit—such as being awarded the 
Broad Prize for Urban Education, there are murmurs and shouts of discontent about the lack of 
community and parental input in shaping reform? With the abolition of all public forums for 
citizen input into school system decisions, new advocacy groups have formed and existing ones 
have become more active.* The advocacy group, New York City Public School Parents, petitioned 
the Broad Foundation, which had already donated more than $15 million to the DOE in support 
of various initiatives, not to award its prize to New York City, citing parents’ concerns about lack 
of access and their questions about school accountability (New York City Public School Parents, 
2007). 

Parent groups have protested at meetings where Chancellor Klein has given speeches on 
issues like the cell phone ban; the test-based third-, fifth-, seventh-, and eighth-grade retention 
policy; and the school report cards. One protesting group, Class Size Matters, has advocated 
for more funding to reduce class sizes and has rejected the legitimacy of DOE-generated parent 
surveys, generating an independent parental survey whose results were much more critical of the 
administration (Haimson, 2008). Teachers and school leaders have also organized around issues 
they would like changed. For example, the New York Collective of Radical Educators advocates 
for social justice, including opposing the use of high-stakes testing, and the current form of 
mayoral control (http://www.nycore.org/statements.html). 

State and local policymakers are starting to look more closely at the systemic privatiza- 
tion of NYC DOE (Hernandez, 2009). In 2008, New York state comptroller Thomas Di- 
Napoli announced that his office would begin auditing the DOE’s no-bid contracting (Med- 
ina, 2008b). Public advocate Betsy Gotbaum, an independently elected member of New 





3This award is given to the most improved urban school system that is funded by venture philanthropist Eli Broad. 
He favors corporate-style management and competition to improve urban schools. 

4New York City has had a complicated school governance history, with community boards being the only elected 
bodies. These were abolished under mayoral control. For a discussion of the evolution of school governance, see Hemphill 
(2009). 
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York City’s government, formed the Commission on School Governance to examine the 
grievances of parents and community members in advance of a mandated reauthorization vote 
for the mayoral control law in 2009 (Robinson, 2008). It commissioned papers, held hear- 
ings, and has produced recommendations to rebalance mayoral control with opportunities for 
substantive community input and external oversight of DOE fiscal and educational policies 
(http://pubadvocate.nyc.gov/advocacy/schools/index.html). 

Whatever the law’s fate, many believe that the privatization reforms of the Bloomberg/Klein 
administration will continue for some time. Feigenbaum and Henig (1994) wrote, 


Not only do the interests of constituencies often change with privatization, but once in place, the 
new policies may engender new groups as well as behavior, groups endowed with powers that leave 
them entrenched and sometimes impossible to remove... . This is, of course, what the opponents of 
privatization fear and what the advocates of privatization are counting on. (p. 208) 


If this argument plays out, the privatization that has been put into motion in NYC DOE is 
unlikely to cease, even under new leadership. 

The contemporary expansion of privatization in New York City and other urban systems also 
suggests new roles for educational leaders and, in many ways, introduces new leadership forms 
into public education. Educational leadership research has only begun to examine the most recent 
privatization expansion, and existing work often neglects the politics of educational privatization. 
In addition, traditional, university-based educational leadership certification programs, in general, 
have not kept pace with the changes underway in many urban school systems, leaving graduates 
of such programs potentially unprepared to deal with this new institutional terrain (see Cibulka, 
2009/this issue). Privatization in many urban school systems requires school leaders to negotiate 
and oversee contracts with private sector actors—whether partners, managers, or profit seekers— 
often adding an additional “boss” or external organization to which they are held accountable 
in market terms (see Shipps & White, 2009/this issue; DiMartino, 2009b). Responding to this 
growing market, leading business schools have hosted conferences and case competitions on 
educational leadership and management (Haas Education Leadership Club, 2009; Yale School 
of Management Club, 2009). In 2008, the KIPP School Leadership Program held its summer 
institute at New York University’s Stern School of Business. 

If New York, New Orleans, Philadelphia, and Washington, DC, are harbingers of the growth of 
systemic privatization in urban school districts, universities should take note. Teacher education 
and leadership preparation programs must attend to the new policy terrain in which teachers 
and principals work. In some ways, the emergence of private providers of public schooling 
is diminishing the desirability of educators trained in traditional university-based programs. In 
fact, school leadership preparation programs and workshops can now be found within several 
university-based business schools. 

For example, a new Brooklyn charter school called Ascend is an amalgamation of many of 
these actors. It is a CMO but licenses aspects of its school design from SABIS, Inc., a for-profit 
EMO. A 2009 job posting for a director of operations revealed that it sought a candidate with 
the following pairing of characteristics: a CPA or MBA degree or equivalent experience and 
experience working in an educational organization or a school, preferably in an urban setting 
(Brownsville Ascend Charter School, 2009). Alumni of Teach For America, Education Pioneers, 
The New Teacher Project, or New Leaders for New Schools, all alternative teacher and leadership 
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preparation programs, were especially encouraged to apply. In this example, several of the types 
of privatization actors identified in our typology come together in the context of one institution. 
For adherents to traditional forms of public education, this should signal how complex changes 
in public education leadership and management have become and how closely they must attend 
to the multiple dimensions of this new educational environment. Simply arguing to preserve the 
status quo is unlikely to convince policymakers oriented to systemic privatization. 

A final implication of this analysis that requires much more exploration is the relationship 
between educational privatization and educational equity, especially as it pertains to race and 
social class. A question requiring much more consideration is why so many policy makers agree 
privatization is the remedy for struggling urban school systems but is not required in predom- 
inantly White and middle-class or wealthy school districts. It is also important to examine the 
role privatization and choice reforms might play in reinforcing racial and social class segrega- 
tion within and across school districts, and what the characteristics are of schools operating in 
privatized schooling systems in terms of leadership, curriculum, governance, and teacher quality. 

Exploring these questions does not preclude a critique of the status quo in many urban school 
systems. Although many urban school systems—and the leadership within them—are themselves 
victims of inadequate funding from state legislatures, declining tax revenues, and deteriorating 
infrastructures, they are also perpetrators of educational neglect and corruption. Many troubled 
urban districts can undoubtedly benefit from some of the practices of the private sector as well 
as from highly functional and equitable public systems. A danger is that infatuation with the 
private sector can inflate assessments of the talent, insight, and innovation its members actually 
provide to public education and underestimate the managerial, pedagogical, and social justice 
capacities that already exist within urban public school systems. In addition, because democratic 
and equitable practices are not hallmarks of the private, for-profit sector, privatization across 
multiple dimensions must be closely monitored by public agencies to protect against abuses 
(Sclar, 2000). 


CONCLUSION 


Educational privatization involves the crossing, or at least the blurring of private and public sector 
boundaries, altering, and possibly enhancing educational goals of economic and social opportu- 
nity. Without examining the full range of privatization actors, our understanding of educational 
and institutional arrangements is attenuated, the shift in power relationships becomes opaque, 
and the profound alterations to leadership, teachers’ work, and community participation in demo- 
cratic governance receive insufficient attention. Indeed, when economic models of privatization 
dominate policy discourse and research, the primary concerns are fiscal issues and assessments 
of student achievement based on standardized exams. 

We propose that our typology of educational privatization has at least three benefits for research, 
policy, and practice. First, it facilitates a more nuanced public debate about the appropriate role of 
the private sector in public education by considering the social, political, and economic dimensions 
of privatization. Second, it highlights the potential promises and pitfalls of privatization for urban 
education, especially in terms of democracy, equity, and school quality. Finally, it enables us to 
closely examine how privatization changes the working conditions and educational experiences 
of those closest to it: school leaders, teachers, parents, and students. Privatization may have the 
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potential to revitalize aspects of public schooling, but we know too little about how it might 
also constrain other aspirational characteristics of public schooling, qualities Americans have 
historically hoped that public education will engender. 
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University-based leadership preparation programs no longer enjoy a near monopoly on the right to 
prepare school principals and other administrative leaders, and now compete with a growing number 
of alternative providers. This article utilizes the new institutionalism literature to analyze this shift. 
Practitioners and policymakers demanded reforms beginning in the 1980s. Two alternative state 
responses are discussed—increased regulation alongside deregulation. University-based providers 
have been weakened by value disagreements, inability to monitor quality, and weak research on 
program effectiveness. Strategies are discussed for creating a market of providers that is more 
strongly regulated around quality. 


This article examines developments that have placed colleges and universities with leadership 
preparation programs (LPPs) on the defensive. University-based LPPs no longer enjoy a near 
monopoly on the right to prepare school principals and other administrative leaders in education, 
as was the case as recently as 2 decades ago. Today many new providers of programs have 
emerged. State departments of education or state licensure boards, which establish the rules for 
program approval and for licensure! of school principals and other leadership personnel, have 
relaxed their regulation to facilitate a competitive market of diverse providers. The programs 
most greatly affected by this development primarily prepare principals and superintendents, but 
other roles also have been affected to a lesser degree such as assistant principals and directors of 
instruction. 

Why has this shift occurred? To interpret the developments that are described in this article, I 
draw on the literature from the new institutional analysis. 


A THEORETICAL PERSPECTIVE 


This article uses an institutional lens to interpret these developments, drawing on the contri- 
butions of the new institutionalism literature. Why use this framework? Institutional theory, 
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particularly as it has been developed by sociologists, has been applied quite extensively to 
understand educational organizations. The theory posits that educational organizations operate in 
a highly institutionalized arena where pursuit of goals often is displaced by pursuit of legitimacy. 
Legitimation emerges as a response to conflicting demands emanating from the institutional 
environment, particularly when the technology of the organization makes it difficult to measure 
whether organizations are performing successfully. Measuring student achievement is, of course, 
complicated. Technical ambiguity about measurement of goals and performance also contributes 
to the lack of coordination and control over teaching and learning (Weick, 1976). Further, the 
public’s expectations for schools vary and are disputed. 

Meyer and Rowan (1977, 1978) argued that under conditions of goal uncertainty, many features 
of modern bureaucracies are, in fact, rationalized myths that become institutionalized as rulelike 
understandings about how to organize an enterprise to achieve particular ends. Rowan and Miskel 
(1999) explained that many of these responses cannot be understood as “rational.” Institutional 
theory challenges “models of social and organizational action in which relatively autonomous 
actors are seen as operating with unbounded rationality to pursue their self-interests”” (p. 359). 

Other scholars (e.g., Rowan, 1981, 1982a, 1982b) have applied the new institutionalism to 
educational administration and the politics of education. For example, Crowson, Boyd, and 
Mawhinney (1996) edited a volume whose contributions centered on the implications of the new 
institutionalism for the study of the politics of education and policy analysis. Ogawa (1994) 
applied the concept to school-based management. 

Although institutional theory offers an explanation for why organizations maintain their legit- 
imacy irrespective of goal attainment, the theory also permits us to see the process of institution- 
alization as destabilizing. The logic of legitimation may buffer the organization from responding 
to demands from its environment, thereby threatening its legitimacy (Cibulka, 1995). 

The current loss of confidence in leadership preparation programs can be understood as a 
crisis of legitimacy. The inability of the profession to respond to the expectations in the policy 
community has destabilized this institutional framework. 


Organization of the Analysis 


The article is organized as follows. First, the institutional framework is explained and a brief 
history of its development is provided. Second, the emergence of new expectations for reforms in 
LPPs coming from practitioners and the policy community beginning in the 1980s is discussed, 
along with two alternative state responses: stronger regulatory accountability and/or deregulation 
and markets. The third section of the analysis describes four problems in the professorate that have 
limited its credibility with the policy community and its effectiveness in reforming leadership 
preparation. The concluding section of the paper lays out a possible strategy for the profession to 
restore its credibility and relevance. 





2Barlier versions of the theory ignored the role of individual actors entirely, but more recent developments do 
incorporate perspectives that capture the role of “agency” in shaping institutions. As Rowan and Miskel (1999) explained, 
the new institutionalism in sociology often subordinates actors’ rational action to their pursuit of institutional legitimacy. 
This perspective “sees rational action . . . as a cultural construction in which ends-means sequences and the very calculus 
of rational efficiency are understood as ideological constructs institutionalized in society” (p. 362). 
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THE EMERGENCE OF A NEW INSTITUTIONAL SECTOR 


Educational administration as a discipline with specialized knowledge evolved as part of a new 
institutional sector that was inspired by Progressive ideology, with its commitment to trained 
executive leadership. As it evolved, this institutional framework came to rely on a combination 
of bureaucratic regulation and professional self-regulation through accreditation. 

Meyer and Scott (1983) discussed the concept of institutional sectors. These entail all the 
actors that produce, distribute, consume, and regulate a product or service. Public schools, for 
example, form a population of more than 15,000 producing organizations, but they are embedded 
in a much larger institutional sector consisting of suppliers, numerous consumers (taxpayers, 
parents, businesses, etc.), and regulators. 

Institutional theorists argue that institutional arrangements are created and sustained in several 
ways; they have regulatory, normative, and cognitive roots (Scott, 1995). The origins of LPPs to 
the Progressive reform movement in the early part of the 20th century embraced all three of these 
elements. Progressives were remarkably successful in shaping the institutional arrangements 
of American public schooling still existing today. Tyack (1974) described Progressives’ efforts 
to build a “one-best system.” Many of the reforms created by Progressives were addressed at 
wresting control from urban political machines and laypersons. Progressive reform principles 
were extended more broadly to consolidation of school districts, curricular reforms, age-grading, 
student tracking, and other education reforms. 

One Progressive strategy cutting across many governmental domains, including education, 
was governmental regulation of the actions of private individuals, groups, and organizations to 
protect the public interest (Tarr, 2000). Drawing their inspiration from business and industrial 
bureaucracies, Progressives also favored putting trained executives in charge of professional 
bureaucracies. Administrative Progressives argued that specially trained experts should operate 
these bureaucracies. They were strong advocates of using Frederick Taylor’s “scientific man- 
agement,” with its principles of bureaucratic hierarchy and efficiency, to the tasks of organizing 
and administering America’s public schools (Callahan, 1962). State bureaucracies were to be 
organized along the same principles. 

This ideology contributed to two complementary developments affecting leadership prepa- 
ration in elementary and secondary schools. First, it led directly to the momentum to profes- 
sionalize training of school principals and superintendents. Progressives argued that university- 
trained experts, unlike politicians and parents, would bring specialized knowledge as well as 
a commitment to impartiality and standardization, in the interests of fairness and efficiency. 
With these unifying principles, training programs multiplied in universities to prepare principals 
and superintendents by offering courses, and even master’s degrees, in educational adminis- 
tration. The second development was a state regulatory framework to manage licensure and 
programs. 

The expansion of these training programs in educational administration continued for many 
decades (and, in fact, continues today despite some contraction). Although the University of 
Michigan and Columbia University had developed limited coursework prior to 1900, it was in 
the decades beginning in 1915—-when Progressive ideas had picked up momentum—and extend- 
ing through the 1930s that programs were developed at many universities (Moore, 1964). This 
movement fed on itself as doctorally trained Progressive school administrators established them- 
selves in the public eye as the “certified experts” by conducting large scale surveys, developing 
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curriculum guides, and otherwise erecting the edifice of the new “system” of public education 
(Grant & Murray, 1996). 

Although no accurate record exists of how many of these programs were created, the University 
Council on Educational Administration estimates that today there are approximately 500 that offer 
master’s degrees, in addition to an unknown number that lead to licensure/certification only. The 
multiplication of such programs, as well as their highly variable quality, makes them somewhat 
analogous to the medical profession before the profession closed many poor quality programs in 
the early 20th century (Starr, 1982). 

Of course, Progressives were convinced that their reform nostrums would raise educational and 
governmental standards, so this raises the question of why so many LPPs of variable quality were 
permitted to exist. One answer is that, like the development of LPPs, the expansion and refinement 
of this state regulatory framework emerged over a period of decades. In our decentralized political 
system, where education is by law and tradition a state and local responsibility, states varied widely 
in their regulatory policies; adoption of regulations varied in timing and intensity from state to 
state, depending on state political cultures, leadership, and tax resources. By the 1930s many 
states were licensing this new class of administrative professionals. (State licensure of teachers 
generally preceded these developments; see Imig & Imig, 2008.) 

States also began to use their regulatory authority to approve the LPPs offered in higher 
education. In some states, program approval conferred on the institution the right to recommend 
licensure to the state upon the candidate’s completion of an approved program of study. However, 
states varied in whether they linked licensure to graduation from an approved program of study 
at a university. 

Although the pattern was a gradual one, it was one of institution building, and newfound 
legitimacy for the LPPs that flowed from these arrangements. By the mid-1950s, more than 80% 
of the states required completion of some graduate coursework for administrative licensure. More 
than half the states required completion of a master’s degree (McCarthy, 1999a). 

This period, which extended for another several decades into the 1980s, might be regarded 
as the high point in the credence paid to the Progressive model, as well as its other policy 
achievements. LPPs became part of a complex institutional framework in which approved pro- 
grams enjoyed a clientele because they alone had the authority to offer programs that meet state 
licensure standards. 

At the same time it is important to document why the state regulatory framework for LPPs was 
relatively weak and made it vulnerable to attacks in the 1980s and thereafter. First, educational 
requirements for licensure, training, and program approval were never universal across all states. 
Coordinating efforts among the states during these decades were weak. It was not until the 
1960s that the National Association of State Directors of Teacher Education and Certification 
(NASDTEC) recommended common standards (Mackey, McHenry, & Einreinhofer, 2003) for 
teacher education and other programs. Even then, these voluntary standards were so vague as 
to allow any state to make the claim that it was meeting the national standards. State education 
agencies also were understaffed, and poor salaries made it difficult for them to recruit highly 
competent personnel, further eroding any pretense to rigorous bureaucratic regulation of program 
quality. 

During this period state standards reflected the philosophy that states should set only minimum 
standards for licensure, or for whatever/whomever they were regulating. Regulatory policies (and 
the institutional structures surrounding them) purported to provide quality assurance; they were 
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designed to assure the public that the quality of a program was at least minimally adequate, 
thereby assuring that its graduates would “do no harm.” In the case of educator preparation, 
the criterion normally was that graduates of LPPs would not harm children. This “do no harm” 
criterion still is widely applied in setting the bar for pass rates on state licensure exams. 

Regulatory regimes that focus on minimums can regulate through efforts to assure compliance 
with mandates that focus on program inputs. Accordingly, program approval processes generally 
were confined to paper reviews of courses or the curriculum offered by institutions of higher 
education. These bear little direct relationship to how the program actually performs, much less 
how well its candidates are prepared. 

Third, notwithstanding the aforementioned, state regulatory oversight over programs in higher 
education has tended to be especially limited. Whatever successes Progressives had in extending 
state regulation of elementary and secondary schools in the name of equity, quality, and efficiency, 
their regulation of higher education was less, perhaps reflecting deference to the presumption of 
academic freedom. State coordinating authorities for higher education did not exist in many states 
until recent decades; the federal government did not address this void until the 1972 amendments 
of the Higher Education Act of 1964 (Thelin, 2004). Generally, higher education has enjoyed a 
presumption of significant autonomy from state oversight, and to a degree this orientation has 
shielded LPPs from rigorous bureaucratic regulation. 


DEMANDS FOR REFORM OF LPPs 


The education reform movement of the 1980s opened a debate about the adequate performance of 
American students. In doing so, it also raised questions about whether teachers were adequately 
prepared to help all students learn at high levels and whether school administrators were being 
prepared to help schools move to high performance. 

Initially, LPPs did not receive much attention in the education reform movement from state 
regulatory agencies, governors, or state legislatures. Most attention was given to reform of teacher 
preparation (e.g., Holmes Group, 1986). However, states heard complaints from school districts 
about the quality of the graduates coming out of these LPPs and from the graduates of programs 
who were dissatisfied with their preparation programs. 

The effectiveness of state regulation of teacher preparation programs through testing also 
began to be debated in the 1980s (McCarthy, 1990). Weak state regulation had been justified 
on the premise that the state’s role is merely to prevent “no harm” to students. However, this 
approach led to low expectations for students and schools, where high rates of failures and drop 
outs were tolerated. Also, it ignored the quality of the teachers these programs produced. 

Meyer and Scott (1983) provided a helpful way of interpreting the reason why the older regu- 
latory framework proved inadequate as the performance of American schools came under greater 
scrutiny in the 1980s. They posited that organizations are embedded in both institutional and 
technical environments. The former reward conformity, whereas the latter reward performance. 
Weak technical environments encourage institutions to engage in compliance and convergence 
around practices that may make them legitimate but that may bear no relationship to goal achieve- 
ment or even contribute to displacement of goals (Rowan & Miskel, 1999, p. 365). It follows 
that regulatory policies that focus mainly on eliciting compliance rather than performance will 
reinforce these tendencies. In these cases, regulation may serve largely a legitimation function 
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by holding institutions accountable for standards of quality that are decoupled from evidence of 
actual effectiveness. 

If state regulation was ineffective, however, what path to reform should be followed? The 
1980s reform movement spawned two very different pathways to reform, as these applied to 
K-12 schools and to higher education’s role in preparing professionals for schools. One approach 
sought to reform regulation itself, the second to weaken or dismantle it. 


One State Response: Intensification of Regulation 


One reform impulse was to strengthen those state policies that would build a technical environment 
focused on performance of students and school personnel. The problem for state policymakers 
was not to guarantee a floor beneath which no one could fall, but to raise the ceiling. “Systemic 
reform” was advocated (Smith & O’Day, 1991) including development of standards for students. 
States increased their regulatory oversight of school districts by developing standards for pre- 
K-12 students. Although state program standards for higher education programs had preceded 
pre-K-12 student standards, the era of student standards-setting led to demands to align higher 
education preparation program standards more closely with P-12 education. 

Thus, even education reformers in the 1980s quickly turned away from minimal compliance 
approaches such as increasing graduation requirements and other inputs. They began to focus on 
higher levels of student performance, such as students’ ability to demonstrate critical thinking, 
and concomitantly the reforms that showed promise of creating high student performance. 

Reform of licensure and preparation programs (Hart & Pounder, 1999; Imig & Imig, 2008) 
also was part of this reform agenda. In 1987 the Council of Chief State School Leaders created 
the Interstate New Teachers Assessment and Support Consortium. Although the group’s function 
was focused on improving state licensure standards, it also turned its attention to helping states 
develop model preparation programs. An off-shoot of that group was the Interstate School Leaders 
Licensure Consortium, focusing on five key state policy levers: professional certification, profes- 
sional standards and assessments, professional preparation, professional development, and state 
reporting and accountability for administrator quality. Because participation in the consortium is 
voluntary, only about half the states have participated at any one time. Nonetheless, this resource 
began to turn the conversation toward alignment of policies, outcome-based approaches to setting 
standards, and other indicators of improved quality. In 2003-04, 37 states required assessments 
of candidates, the most popular being the School Leader Licensure Assessment offered by the 
Educational Testing Service (Sanders & Simpson, 2005).3 After a decade of effort in 2003-04, 
39 states required a candidate to complete a preparation program for licensure, and 25 states 
required internships as a component of that program. At the time, 39 states required master’s 
degrees (Sanders & Simpson, 2005). 

Some states looked to foundations for support in redesigning their policies. The Wallace 
Foundation created the State Action for Education Leadership Project in 2001 to focus on reform 
of policies and regulations in 15 states. 





3The adoption by states of national assessments for licensure has created incentives for LPPs to align their programs 
to the knowledge and skills contained in the assessments. This was another way in which LPPs were influenced by the 
professionalization agenda. 
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National accreditation also has been a tool to strengthen the regulatory framework 
surrounding LPPs. Unlike state regulation, accreditation represents the profession’s at- 
tempts to regulate itself. The National Council for the Accreditation of Teacher Education 
(NCATE) was created in 1954. Five groups were instrumental in the creation of NCATE 
(http://www.ncate.org/public/aboutNCATE.asp): the American Association of Colleges for 
Teacher Education, NASDTEC, the National Education Association, the Council of Chief State 
School Officers (CCSSO), and the National School Boards Association. 

Despite NCATE’s existence dating to the 1950s, it was not until the 1990s that it began to work 
closely with specialty area associations (SPAs) to set standards for individual programs such as 
leadership. SPAs are membership associations consisting primarily of practitioners, with some 
academics. Although many of these associations had developed program standards, most states 
used their own standards. 

In the program area of leadership preparation, no standards were available until they were 
developed by Interstate School Leaders Licensure Consortium under the leadership of CCSSO 
in 1996. The new standards addressed “the knowledge, skills, and attributes required to lead and 
manage an educational enterprise centered on teaching and learning” (Hart & Pounder, 1999). 
These were initially conceived as licensure standards, but then were adapted for program approval 
under the auspices of the National Policy Board on Educational Administration.‘ 

National accreditation remains voluntary in most states. Nevertheless, NCATE became more 
influential as it developed partnerships with states in the 1990s, which led to better alignment 
between national accreditation and state program approval. Also, by 2002 NCATE had moved 
away from course or curriculum standards to outcome-based standards. LPPs were required to 
develop assessments that demonstrate candidate performance aligned with those standards. In 
2004 NCATE brought the process of developing standards under its management and control. 

This gradual consolidation of NCATE’s authority led more states to utilize NCATE’s national 
program approval process, or to align their program approval policies with NCATE standards, 
even if they maintained separate program reviews. By 2008, 29 states had aligned their program 
reviews with NCATE processes and standards, and an increasing number of these began to use 
the NCATE program review process. 

In sum, NCATE, working with other professional associations such as CCSSO, has helped 
states strengthen their regulation of LPPs. Its focus on candidate outcomes has begun to move the 
regulatory framework surrounding LPPs toward performance rather than the minimal standards 
and “do no harm” policies that became discredited by the 1980s. 


A Second State Response: Deregulation and Markets 


Many states also responded to complaints about leadership preparation programs in the 1980s 
in a very different way. They deregulated their policies and created a competitive market. The 
election of Ronald Reagan in 1980 had ushered in a period of anti-government sentiment at 
all levels of the political system. The “Reagan revolution” included weakening and dismantling 
regulatory regimes wherever they existed in favor of the so-called self-regulation of the market. 


4The Educational Leadership Constituent Council is an arm of the Board. It consists of three NCATE member orga- 
nizations (SPAs): the Association for Supervision and Curriculum Development, the National Association of Elementary 
School Principals, and the National Association of Secondary School Principals. 
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The 1980s deregulatory fever at the state level prompted the National Policy Board on Education 
Administration (Thompson, 1999) to develop qualifications that candidates for alternative cre- 
dentials should meet, to guide states. At a national level, alternative routes had been endorsed by 
the influential A Nation at Risk report early in the decade, which encouraged states to move in 
this direction (Firestone, 1990). ‘ 

The last 2 decades have witnessed many efforts to deregulate state and local educational 
services and to introduce various consumer choices and market competition in education (Cibulka 
& Boyd, 2003). As privatization efforts expanded in P-12 education, this created a demand for 
administrators for charter schools and contract schools. 

One approach to meeting this demand was deregulation—relaxation or elimination of licensure 
requirements. Michigan and South Dakota have eliminated licensure requirements altogether. By 
the 1990s it was common to recruit urban superintendents from nontraditional backgrounds such 
as governors, former military officials, and business leaders (Cuban & Usdan, 2003). By 2009, 
only a few states required master’s degrees for licensure as a principal (http://www.nasdtec.org). 

Another approach authorized alternative providers of preparation programs. Elmore (2008) 
criticized state policies as reinforcing the status quo and being part of a cartel with higher 
education and local school districts, although he acknowledged that more than one fourth of 
the states now have some form of alternative certification for school leaders. Advocates of 
markets (e.g., McGuinn, 2006) argue that markets are more likely to produce the supply of 
professionals needed and to improve diversity. One fourths of states now permit alternative 
licensure of education leaders, either autonomous from higher education or in collaboration with 
it (Elmore, 2008; McCarthy & Forsyth, 2009). 

A number of urban school districts now offer their own training programs for school leaders, 
bypassing university-based programs altogether. New providers, some with venture capital sup- 
port, have entered the market. New Leaders for New Schools began in 2001 and now operates in 
nine urban districts. Like many alternative providers it has a more focused mission than traditional 
LPPs. It seeks to produce school leaders committed to high levels of learning and achievement for 
every child. Although still relatively small (431 graduates in 2006), the program has ambitions 
to address the leadership challenge “at scale” (New Leaders for New Schools, 2008). 

The Broad Foundation’s Residency Program recruits individuals to work in management 
positions in urban school systems and charter school management organizations. The Foundation 
also supports a Superintendents Academy to prepare CEOs and senior executives from business, 
nonprofit, military, government and education backgrounds to lead urban public school systems. 
Such initiatives would not be possible without changes in state (and local) program regulations, 
licensure rules, and civil service policies to recruit nontraditional individuals into these positions. 

New private-sector models of schools of education are emerging to challenge colleges and 
universities. The HTH Graduate School of Education in San Diego offers masters degrees in 
school leadership for individuals who wish to lead a small ‘“<nnovative” school. Also, schools of 
education are losing their monopoly of training programs within universities. Rotherham (2008) 
cites as an example the Rice Education Entrepreneurship Program operating entirely within a 
business school. 

Taken together, these deregulatory and market responses sanctioned by states operate in tension 
with the effort to strengthen LPPs through greater regulation. Although most states continue to 
take both paths, it is clear that momentum has increased for this second pathway. 
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REFORM EFFORTS WITHIN THE PROFESSORIATE 
AND THE ACADEMY 


It is unlikely that deregulation and market responses would have grown if the professoriate were 
able to gain more respect from policymakers and the practice community. The field suffers from 
a number of long-standing problems including a lack of consensus on the knowledge base, the 
persistence of an academic program model, the inability of higher education to police itself, and 
a lack of research on effective programs. Each of these problems is discussed briefly. 


Lack of Consensus on the Knowledge Base 


Many scholars have chronicled the inability of the professoriate in educational administration 
to develop a knowledge base that is widely agreed upon. In the early decades after LPPs were 
created, the faculty members of such programs typically were drawn from the field of practice 
(almost exclusively White men) and had little or no formal training in their fields. As recently 
as the end of World War II, leadership preparation for principals and superintendents was highly 
practical and lacked any pretense to a scholarly base (Cooper & Boyd, 1987; Griffiths, 1959). 
LPPs were staffed by practitioners who dispensed experiential wisdom (McCarthy & Forsyth, 
2009). Murphy (1992) referred to this as the prescriptive era. It was not until the so-called theory 
movement began in the late 1940s grounded in behavioral science that preparation programs 
and their professors began to aspire to resemble an established academic discipline. Scholars at 
leading universities attempted to expand the knowledge base by infusing social science knowledge 
in the new discipline (Griffiths, 1964). Formal training programs began to be developed for the 
professoriate, and scholarship began to emerge, although little of it focused on what constitutes 
successful practice in this field. 

However, after the theory movement’s heyday between 1947 and 1957, controversy ensued over 
its philosophical assumptions. By the time it gained momentum in educational administration, 
its assumptions already were being questioned by human relations theorists. Also, as Murphy 
(1992) explained, the hypothetico-deductive approach came under increasing criticism. Among 
its earliest and most articulate critics was Greenfield (Greenfield & Ribbins, 1993). 

The University Council of Educational Administration (UCEA), which represents the pro- 
fessoriate at research universities, has played a leadership role in promoting reform since its 
inception in 1956 by 34 leading universities. However, UCEA encountered strong resistance 
when it attempted to codify the knowledge base after the collapse of the theory movement. Its 
1989 effort (Forsyth & Tellerrico, 1993) to define a knowledge base for preparing urban prin- 
cipals failed to gain traction. UCEA also initiated the knowledge-base project in 1992. Critics 
subjected the findings (Hoy, Astuto, & Forsyth, 1996) to withering criticism. Donmoyer, Imber, 
and Schuerich (1995) asserted that the findings reflected a functionalist paradigm, which gave 
insufficient attention to critical perspectives in the curriculum. 


Persistence of an Academic Program Model 


Another layer of debate has persisted on whether to organize the curriculum around technical- 
rational knowledge or around problems of practice, following the work of Schon (1987) and 
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others (Jacobson, 1990). Forsyth and Murphy (1999) argued for a professional approach to 
training grounded in practice, distinct from programs oriented primarily to teaching theory with 
little actual opportunities to apply learning. 

Within education, in the evolution from normal schools to education schools, the academic 
model won out (Labaree, 2004). Despite efforts by reformers such as the Holmes Group (1986), 
schools of education have not created a sufficient supply of professional development schools 
that might serve as the analogue of teaching hospitals, nor have they restructured themselves 
internally to facilitate this focus on practice. Leadership programs also reflect this academic 
model. The profession’s contributions to building a stronger technical knowledge base have been 
organized primarily around social science disciplinary knowledge (Forsyth & Murphy, 1999) 
rather than practice knowledge linked to an evidentiary base on program outcomes. Although the 
Danforth Foundation supported program redesign at 92 universities in the early 1990s, focusing 
on incorporating greater practice knowledge in LPPs (McCarthy, 1999b), that effort did not lead 
to widespread adoption elsewhere. 

There were powerful institutional pressures to emphasize an academic rather than professional 
approach. The discipline of education has held a particularly tenuous place in research universities, 
where teacher and leader preparation were never a core mission. Education competes with more 
established academic disciplines and professions. The reward structure has favored specialization 
and research rather than clinical preparation of practitioners. 

Given the failure of LPPs to demonstrate significant commitment to reforming themselves, a 
chorus of critical studies and reports have been issued in recent years (e.g., Bottoms & O’ Neill, 
2001; Broad Foundation and Fordham Institute, 2003; Hess, 2003; Levine, 2005). 


Inability to Police Itself 


Although many reform proposals have emanated from within the profession since the 1980s, most 
have had difficulty being implemented. An influential report, Leaders for America’s Schools 
(National Commission on Excellence in Educational Administration, 1987), generated much 
discussion about how to reform LPPs. The Danforth Foundation supported program redesign 
efforts in a number of universities, and under the leadership of Professor David Clark at the 
University of Virginia, supported the creation of the National Policy Board on Educational 
Administration (NPBEA). 

NPBEA represented the collective voice of ten organizations and brought together the profes- 
soriate and practice communities. In 1989 it issued a report, Improving the Preparation of School 
Administrators: The Reform Agenda, which recommended a common curriculum, national ac- 
creditation, and national certification. Some of its recommendations generated controversy. Pro- 
posals to reduce the number of LPPs and make them more selective were not well received by 
many programs and faculty. The professoriate showed no ability to police itself. 

Other reform efforts emanated primarily outside the academy but within the profession. 
Practitioner groups such as the National Association of Secondary School Principals and the 
National Association of Elementary School Principals became frustrated at “a mismatch between 


SNPBEA had more success in the development of national licensure and program standards, working with CCSSO 
and NCATE, as was discussed earlier in the article. Higher education representatives also were participants in those 
efforts. 
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preparation programs and job requirements.” Consequently, they created the National Commis- 
sion on the Principalship. 

A number of reviews have found little evidence of true innovation in programs, apart from 
changing labels of courses. Even areas in which there is broad agreement for improvement 
have demonstrated only incremental progress. Clinical experiences have undergone only modest 
expansion of clinical experiences. LPPs have been reluctant to deny admission to teachers who 
enroll in their programs primarily for salary advancement and who have little intention to become 
principals. (These studies are summarized in McCarthy & Forsyth, 2009.) 

Some programs continue to be staffed primarily by part-time adjunct faculty with practitioner 
backgrounds but little scholarly training; in this respect they are only marginally different from 
programs that existed in the 1940s. Some of these problems may be rooted in institutional 
pressures on LPPs from provosts to keep enrollments high because this increases tuition revenue. 
At the same time, hiring part-time faculty and avoiding expensive clinical supervision keeps 
program costs low. 


Lack of Research on Effective Programs 


Until recently, little research has been conducted on effectiveness of reforms to traditional pro- 
grams (Orr & Pounder, 2008). As late as 1989, a volume on reform of school administration 
(Hannaway & Crowson, 1989) focused entirely on reforming practice rather than addressing 
reform of LPPs, as though there were no connection between the quality of training and the 
resultant quality of practice. Perhaps such research lacks prestige and is not easily published. 
The research void also reflects little consensus on what the knowledge base should be and what 
competencies practitioners should demonstrate. 

More recently, the policy community has demanded evidence that teacher and leader programs 
demonstrate a positive impact on P-12 student learning. Some alternative programs such as New 
Leaders for New Schools are developing these data. So far very little research has come from the 
education leadership field to address this demand for accountability.® 

Given these four problems—a disputed knowledge base, disagreement on whether to structure 
LPPs as academic or professional programs, inability to police itself, and the weak interest in 
building an evidentiary research base on effective programs and practice—the professoriate has 
been unable to position itself as a leading voice favoring reform. 


CONCLUSION 


This article has reviewed the long-standing efforts to create high quality LPPs. These reform 
initiatives reach back to Progressives early in the last century. Support for the institutional 
framework inspired by Progressives has eroded in recent decades. This framework supported 


®It is also the case that research conducted by economists and policy researchers has focused mainly on whether 
traditional teacher preparation programs and routes to licensure make any difference in student achievement outcomes 
(Kane, Rockoff, & Staiger, 2006). New Leaders for New Schools contracted with the Rand Corporation to evaluate the 
effectiveness of its graduates on improving student achievement. 
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preparation programs offered by colleges and universities, regulated by the states. However, 
policy expectations have shifted away from an institutional environment built around compliance 
to a technical environment focused on productivity. Most states have moved in two directions 
at once, although in varying degrees— intensification of regulation with a shift in focus toward 
candidate outcomes, and authorizing deregulation and alternative providers. This second state 
response has been expanding. 

Despite the purported focus on productivity in the current policy debates on rebuilding effective 
LPPs, neither traditional programs nor new providers are building their training programs on a 
strong research base concerning the characteristics of effective training and practice. The path 
forward requires the development of a much stronger technical knowledge base on these matters. 
Only with this evidentiary base will there be the basis for creating and sustaining effective 
programs focused on performance, building effective government regulation, and assuring strong 
self-regulation by the profession. 

This research needs to be targeted on critical needs in the field. Goodlad (1984) argued several 
decades ago that P-12 education and teacher preparation should be transformed simultaneously. 
The same principle holds for leadership preparation. An example is preparation of principals to 
raise student achievement in low-performing schools. LPPs should align their programs more 
closely with the needs of school systems and build a research base that informs effective leadership 
preparation grounded in problems of practice. 

Recently, more attention has been paid by the profession to documenting the positive effects 
of innovative LPPs on principal practices and school improvement (e.g., Orr, 2008). UCEA 
convened an Evaluation Research Task Force to identify characteristics of successful programs 
that have an impact on practice. This group works collaboratively with other partners such as the 
American Educational Research Association Special Interest Group “Leading and Teaching in 
Educational Leadership,” the National Council of Professors of Educational Administration, and 
the New York Leadership Academy. 

These research initiatives offer hope that there will emerge a stronger technical environment 
in this institutional arena, led by researchers and LPP reformers. The goal should be to build 
a market of providers that is informed by, and competes around objective measures of quality 
grounded in research. 

The educational leadership professorate can reverse its declining position within this mixed 
market of providers. To accomplish this though, it must first develop a visionary research strat- 
egy. Second, it must demonstrate a commitment to self-regulation and the elimination of weak 
programs. Finally, it must improve the alignment of its programs with the needs of states and 
school districts, as well as the expectations of accreditors. These several strategies, if pursued 
in concert, show promise of restoring the stature of higher education programs. Ironically, it is 
leadership that now is required. 
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Introduction to the Special Issue on Alternate Assessments 
Based on Modified Academic Achievement Standards: New 
Policy, New Practices, and Persistent Challenges 


Ryan J. Kettler and Stephen N. Elliott 
Learning Sciences Institute, Peabody College at Vanderbilt University 


This introduction to the special issue titled Alternate Assessments Based on Modified Academic 
Achievement Standards: New Policy, New Practices, and Persistent Challenges addresses the federal 
policy introducing the new alternate assessment for students with persistent academic difficulties, as 
well as related implementation issues that will be more thoroughly considered throughout the journal. 
Three guidelines are identified within the policy for alternate assessments based on modified academic 
achievement standards (AA-MASs), including that (a) a state’s grade-level academic content standards 
cannot be modified for an AA-MAS, (b) a state’s general test can be modified for an AA-MAS, and 
(c) a state’s achievement standards can be modified for an AA-MAS so long as they remain on grade 
level. This article introduces key issues including identification of students eligible for an AA-MAS, 
the degree of modification that can be applied to develop an AA-MAS, and the current state of 
AA-MAS development across the nation. The article concludes with overviews of each contribution 
in the journal. 


The final regulations for the No Child Left Behind (NCLB) Act of 2001 indicate that a small 
group of students with disabilities are allowed to show proficiency through an alternate assessment 
based on modified academic achievement standards (AA-MAS; U.S. Department of Education 
[ED], 2007a, 2007b). These students, who may constitute up to 2% of all who are reported 
proficient within a school, can take a version of the regular assessment test with modifications. 
Modifications are changes to a test’s content or item format. The regulations strongly emphasize 
that although modifications may make a test easier, out-of-level (i.e., below grade level) testing 
is not acceptable, leaving developers and users of these AA-MASs to determine at which point a 
test has become too easy to be within the intended level. Modifications to large-scale achievement 
tests, like testing accommodations, are intended to facilitate access to the assessment for students 
with special needs, so that their scores can be meaningfully compared with the scores of students 
who take the standard test. If this can be accomplished, better assessment and accountability for 
students with disabilities would appear to be the result. 


ee eee eee ee ee eee 
Correspondence should be sent to Ryan J. Kettler, 410E Wyatt Center, Peabody #59, 230 Appleton Place, Nashville, 
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on grade-level content standards and, therefore, must be assessed with a measure that is 
also based on grade-level content standards, although the assessment may be less difficult 
than the general assessment. The content standards are not modified, but the achievement 
expectations are less difficult than those on the general test. This means that the same 
content is covered in the test, but with less difficult questions overall.”(ED, 2007a, p. 20, 
D-1) 

e “.. alignment with grade-level content standards is the foundation of an alternate assess- 
ment based on modified academic achievement standards. An alternate assessment based 
on modified academic achievement standards must cover the same grade-level content as 
the general assessment.” (ED, 2007a, p. 21,-D-2) 

e “A State may modify an existing assessment or develop a new assessment.” (ED, 2007a, 
p. 24, D-8) 

e “Some States have suggested replacing the most difficult items on the general test with 
simpler items appropriate for the grade level, while retaining the same coverage of the 
content standards. Others have suggested modifying the same items that appear on the 
grade-level assessment by simplifying the language of the item or eliminating a distracter 
in multiple-choice items (e.g., having 3 options to choose from, instead of 4).” (ED, 2007a, 
p. 25, D-12) 

e “an alternate assessment based on modified academic achievement standards must be 
aligned with State content standards in the same manner as the general assessment ...” 
(ED, 2007a, p. 26, D-12) 

e “Built on grade-level content but with easier items.” (ED, 2007a, p. 51, Appendix A) 


Finally, with regard to the topic of modified achievement standards the ED guidance document 
makes the following points: 


e “A modified academic achievement standard is an expectation of performance that is 
challenging for eligible students, but is less difficult than a grade-level academic achieve- 
ment standard. Modified academic achievement standards must be aligned with a State’s 
academic content standards for the grade in which a student is enrolled. Thus, only the 
academic achievement standards are modified, not the content standards on which those 
modified academic achievement standards are based. Although the assessment and modi- 
fied academic achievement standards for a particular grade must be challenging for eligible 
students, they may be less difficult when compared with the general test and grade-level 
academic achievement standards... . The characteristics of modified academic achieve- 
ment standards are the same as those described in the Title I assessment regulations for 
grade-level achievement standards. That is, they must be aligned with the State’s academic 
content standards; describe at least three levels of achievement; include descriptions of 
the competencies associated with each achievement level; and include assessment scores 
(cut scores) that differentiate among the achievement levels and describe the rationale and 
procedures used to determine each achievement level.” (ED, p. 14, B-1) 

e “Modified academic achievement standards are not based on academic content stan- 
dards that have been modified or restricted. Alternate academic achievement standards, 
on the other hand, are based on a very limited sample of content that is linked to 
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grade-level content standards but may not fully, represent grade-level content and may 
include substantially simplified content.” (ED, 2007a, p. 15, B-2) 

° “Proficient performance on an alternate assessment based on modified academic achieve- 
ment standards... is expected to represent understanding of grade-level content based on 
a less rigorous assessment.” (ED, 2007a p. 21, D-2) 

e “An alternate assessment based on modified academic achievement standards must have at 
least three achievement levels. If a State’s general assessment has six achievement levels, 
the alternate assessment based on modified academic achievement standards does not need 
to have all six achievement levels. In such cases, decreasing the number of achievement 
levels to three, instead of six, would allow the design of a test with fewer items, while 
covering the same grade-level content standards as the general assessment.” (ED, 2007a, 
p. 23, D-5) 

e “...an accurate and meaningful measure of their achievement will require a different 
definition of proficiency in the form of a more appropriate test and related academic 
achievement standards. Setting a lower cut score on the general assessment does nothing 
to make the test more accessible or understandable.” (ED, p. 24, D-9) 


We believe this collective guidance leads to three important conclusions from the stand point 
of an alternate assessment accountability system: 


1. You cannot modify a state’s grade-level academic content standards. In other words, the same 
content standards must be used to guide instruction and the development of an AA-MAS. 

2. Youcan modify a state’s grade-level achievement test for students who qualify for an AA-MAS. 

3. You must modify a state’s grade-level academic achievement standards for any modified 
alternate assessment that is used. 


Figure | provides an integrated summary of the key elements of an AA-MAS assessment and 
accountability system and reiterates our main conclusions about key elements. Let’s now turn to 
several of the implementation issues that currently are challenging educators and test developers. 


WHICH STUDENTS WILL BENEFIT FROM AN AA-MAS? 


The final regulations allow that an AA-MAS may be used for asubset of students with disabilities 
for whom such an assessment may be more appropriate. The modified standards are intended to 
help meaningfully include in the assessment and accountability system those students for whom 
the regular assessment is too difficult but for whom the alternate assessment based on alternate 
academic achievement standards is too easy. These are students whose disabilities have kept them 
from reaching proficiency, and who are unlikely to reach proficiency within the same timeframe 
as students who take the regular assessment (ED, 2007a). These are students with disabilities 
who can make significant progress, but have not reached grade-level achievement standards even 
after exposure to quality instruction, and who have a reliable record of below-level performance 
on achievement tests even with appropriate accommodations. They learn at a slower rate than 
most students and are considered to have persistent academic difficulties. 

Students determined to meet these criteria may take the AA-MAS, so long as no more than 2% 
of the students within a state show that they are proficient via this test. An exception allows that as 
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FIGURE 1 Overview of the key elements of an Alternate Assessments Based on Modified Academic Achieve- 
ment Standards (AA-MAS) System. Note. PLD = proficiency level descriptor, AYP = adequate yearly progress. 


many as an additional 1% of students in a state could show proficiency, as long as less than 1% of 
a state’s student population shows proficiency on the alternate assessment of alternate academic 
achievement standards. Even with this additional consideration, the law may not allow a large 
enough proportion to account for all students who have disabilities that are very likely to keep 
them from attaining proficiency in a state. Based on a survey of representatives of departments 
of education in five states, the second author of this article determined that between 3% and 9% 
of the total student population might meet these criteria. Educators, legislators, and researchers 
must seek a clearer picture of who the 2% of students eligible for this AA-MAS should be. 


HOW MODIFIED CAN THE TEST BE? 


The AA-MAS is expected to be easier than the regular assessment, but should be challenging 
for students who are eligible, and should reflect grade-level content (ED, 2007a). Because the 
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standards must reflect grade-level content, out-of-level testing is not appropriate. Out-of-level 
testing is the practice of assessing a child with a test that is designed for a lower grade level than 
the one in which he or she receives instruction. During the 2003-2004 academic year, 17 states 
used out-of-level testing for some students for whom the regular assessment was considered too 
difficult (Minnema, Thurlow, & VanGetson, 2004). The, requirement that the AA-MAS cannot 
result in out-of-level testing has put test designers in the position of having to determine how much 
easier an assessment can be made before it can no longer be considered grade level. A number 
of technical issues generated by this new policy confront test developers and state assessment 
directors alike. These include concerns about (a) a test’s content alignment with that of the content 
taught and the content in state standards, (b) the depth of knowledge assessed, and (c) evidence 
for the comparability of the constructs assessed. 

Testing accommodations and modifications are alterations made to a test or testing situation 
that make the test more accessible and appropriate for a group of students. According to Hollen- 
beck, Rozek-Tedesco, and Finzel (2000), appropriate accommodations are identifiable by four 
attributes: 


1. Unchanged constructs—The alterations must not alter the construct being measured. 

2. Individual need—The alterations must be individualized, rather than applied to groups. 

3. Differential effects—The alterations must cause differential effects, such that the students who 
need them benefit in ways that other students would not. 

4. Sameness of inference—The alterations must result in test scores that can be interpreted 
similarly to the interpretations of the regular assessment. 


To the degree that a test alteration reflects these four attributes, it is likely to be considered an 
accommodation. If a test alteration fails to reflect one or more of these attributes, it is likely to be 
considered a modification. Examples of modifications include removing one of three distracters 
from a multiple-choice test, reducing the reading load of an item by simplifying language, adding 
visual cues that make an item easier to understand, or retesting students and using the better of 
multiple results. The field has not yet reached consensus on terminology to refer to an alteration 
that reflects the first, third, and fourth attributes but is applied to a group rather than individualized. 
Such terminology may soon be necessary, as states make alterations to items and tests for the 
group of students that qualify for an AA-MAS, in hopes of measuring unchanged constructs, 
evidenced by differential effects, which are useful for making the same inferences. 


WHAT IS THE CURRENT STATE OF AA-MASS? 


As of June 2007, three states (Kansas, Louisiana, and North Carolina) already had AA-MASs 
in place. Twenty of 36 state leaders who responded to a National Association of State Directors 
of Special Education survey indicated that they were going to adopt an alternate assessment 
for 2% of students, as allowed by the new regulations, and 12 state leaders indicated that they 
were undecided. In addition to determining the degree of modification that is necessary, and who 
should qualify for the modified alternate assessment, these states face a number of practical and 
technical challenges. States must train IEP teams to implement decisions about eligibility for the 
new assessment, must determine how to handle repeated testing within a single school year, and 
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must develop plans for management and reporting of additional data and results. States also have 
to develop and validate the new AA-MASs, write performance level descriptors, and implement 
standard setting procedures for the modified tests. 

At the time of submission, the authors of this special issue were aware of one unpublished 
study of large-scale testing, item modifications, and students with disabilities. This study was 
conducted by Famularo and Russell (2006), and their findings were not conclusive about the 
influence of item modifications on the test scores of students with disabilities. Much larger bodies 
of research exist in the related areas of testing accommodations, universal design, and number 
of multiple-choice item response options. These areas, appropriately conceptualized as issues to 
be addressed when making assessments more accessible for all students, can also be viewed as 
guides in the process to develop AA-MASs. 

For example, one popular item-modification strategy with empirical support involves the 
reduction of the number of answer options to a multiple-choice question. By reducing the number 
of response options, it is assumed that you have reduced the complexity of the decisions to be 
made and reduced the amount of reading, if not the difficulty of the task. Rodriguez’s (2005) meta- 
analysis of 27 studies addressed the question, “What is the optimal number of response options for 
a multiple-choice test?” Using the psychometric criteria of item difficulty, item discrimination, 
and test score reliability, the researcher concluded, 


Three options are optimal for MC items in most settings. Moving from S-option items to 4-option 
items reduces item difficulty by .02, reduces item discrimination by .04, and reduces reliability by 
.035 on average. Moving from 5- to 3-option items reduces item difficulty by .07, does not affect 
item discrimination, and does not affect reliability on average... . Moving from 4- to 3-option items 
reduces item difficulty by .04, increases item discrimination by .03, and increases reliability slightly 
by .02. (p. 10) 


Although many states are using this strategy for item modification for students with disabilities, 
Rodriguez’s (2005) findings indicate that reducing the number of distracters does not harm the 
psychometric properties of the test within the general population but does theoretically reduce 
the reading load of the entire test. Modifications such as this are truly in the spirit of making 
assessments more accessible to students with disabilities. 


AN INTRODUCTION TO OUR COLLEAGUES’ ARTICLES 


The authors invited to contribute to Alternate Assessments Based on Modified Academic Achieve- 
ment Standards: New Policy, New Practices, and Persistent Challenges were identified from a 
group of presenters at the introductory meeting in Washington, DC, on January 16, 2008, for Prin- 
cipal Investigators for General Supervision Enhancement Grants focusing on AA-MASs. They 
are educators and researchers who are motivated to address the aforementioned issues evoked by 
the new regulations regarding the AA-MAS. 

In the first article, “The ‘Two Percent Students’: Considerations and Consequences of Eligiblity 
Decisions,” Naomi Zigmond and Amanda Kloo expand upon their presentation at the introductory 
meeting. The presentation provided a critical look at the new policy and the impact that it might 
have on individual, state, and national levels. 
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In the second article, “The Changing Landscape of Alternate Assessments Based on Modified 
Academic Achievement Standards: An Analysis of Early Adopters of AA-MASs,’ Sheryl] Lazarus 
and Martha Thurlow discuss the current policies that states have in place for AA-MASs. Their 
article includes information on the number of states designing new tests, the methods that are 
being used, and the numbers of students likely to be affected. 

In our third article, Andrew Roach and associates examine instructional aspects of an AA- 
MAS. Their article, “Opportunities and Options for Facilitating and Evaluating Access to the 
General Curriculum for Students With Disabilities,” reviews research on potential indicators of 
access to the general curriculum and opportunity to learn. 

In the fourth article, “Modifying Achievement Test Items: A Theory-Guided and Data-Based 
Approach for Better Measurement of What Students With Disabilities Know,’ along with Peter 
A. Beddow, we describe a systematic process of item modification. This process includes a 
foundation in theory, individual item analysis through statistics and a modification tool, and pilot 
testing in a cognitive lab. 

The fifth article by Karla Egan, Steve Ferrara, Christina Schneider, and Karen Barton, titled 
“Writing Performance Level Descriptors and Setting Performance Standards for Assessments 
of Modified Achievement Standards: The Role of Innovation and Importance of Following 
Conventional Practice,” addresses critical aspects of developing performance level descriptors 
and achievement standards for a modified assessment. As they explain in detail, the process for 
making these alterations will have to incorporate consideration of the new test, as well as the 
student population for which it is designed. 

The final article, “State Perspectives on Implementing, or Choosing Not to Implement, an 
Alternate Assessment Based on Modified Academic Achievement Standards,” by Porter Palmer, 
is a report of a national survey of state special education and assessment directors. The article 
highlights key issues on which state leaders are focusing to make decisions about this optional 
assessment for students with disabilities. 

Susan Weigert, a Program Officer at the ED, and Michael Rodriguez, a psychometrician at the 
University of Minnesota, agreed to read all six articles in this issue and provide comments and 
perspectives. Their comments conclude the issue while concurrently providing a springboard for 
further examination of assessment policies, practice, and research. 

Although this issue on AA-MAS will likely raise as many or more questions as it will answer, 
it represents a step toward determining how these assessments should be used to meaningfully 
include students with disabilities in the educational accountability framework, and ultimately 
in the classroom. We hope you find the articles informative with regard to the new alternate 
assessment, changes in practices, and issues that will likely challenge developers and users of 
AA-MASs. 
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Since the 2001-02 school year, the accountability provisions of the No Child Left Behind Act (NCLB) 
have shaped much of the work of public school teachers and administrators in the United States. NCLB 
explicitly prohibits schools from excluding students with disabilities from the accountability system 
and requires not only participation of all students in statewide accountability assessments but also 
reporting of the results for students with disabilities along with other students and as a disaggregated 
group. From the beginning of these requirements, lawmakers recognized that there would be a small 
group of students with disabilities for whom the regular assessment, even with accommodations, 
would not be appropriate and they authorized states to develop an alternate assessment based on 
alternate achievement standards (AA-AAS) for this group of students. More recently, responding to 
pressures from the field, additional flexibility has been granted to develop an additional alternate 
assessment based on modified grade-level achievement standards (AA-MAS) for students with dis- 
abilities who present with persistent academic difficulties. It is expected that approximately 2% of 
the total student population might be included in this new alternate assessment. This article examines 
the decisions that need to be made by individual states to determine the target population for this new 
alternate assessment and the policy implications of these decisions. 


Since the 2001-02 school year, the accountability provisions of the No Child Left Behind 
Act (NCLB; 2001) have shaped much of the work of public school teachers and administrators 
in the United States. NCLB requires each state to develop content and achievement standards 
in several subjects, to administer tests to measure students’ progress toward these standards, to 
develop targets for performance on these tests, and to impose a series of sanctions on schools 
and districts that do not meet the targets. Together, the standards, assessments, and consequences 
constitute a standards-based accountability system. Many states had such a system in place before 
NCLB took effect, but since 2001-2002, every state in the United States has had to develop and 
implement a standards-based accountability system that meets the requirements of the law. This 
mandate has affected every public school student, every public school, and every district in the 
nation. 

Standards-based reform is not new; it emerged in the early-to-middle 1990s as the primary 
vehicle for focusing schools on achievement and on setting high expectations for teachers and 
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students. In 1989, in response to the poor performance of U.S. students on international com- 
parative assessments (McKnight et al., 1987; Travers & Westbury, 1989), the nation’s governors 
sounded the call for explicit world-class standards to guide educational practice. After much study 
and discussion (National Council on Education Standards and Testing, 1992), content standards 
became a formal requirement of the 1994 reauthorization of the 1965 Elementary and Secondary 
Education Act (Improving America’s Schools Act (IASA), 1994). The 1994 reauthorization re- 
quired that by the 2000-01 school year, each state must have in place a statewide assessment 
system that would serve as the primary means for determining whether schools and districts 
receiving Title 1 funds were making adequate yearly progress toward educating all students to 
high standards (IASA, 1994). 

Accountability for outcomes in education is also not new; it can be traced back more than 
a century to early pay-for-performance agreements in England (Kirst, 1990), although such 
arrangements are not part of most formal educational accountability systems today. Instead, 
outcomes-based accountability has taken a number of other forms in the United States, includ- 
ing minimum competency testing and high school exit examinations in the 1970s, and more 
recently, grade-level promotion tests. Yet the combination of the three elements—higher content 
standards, the use of assessments aimed at measuring how well schools are helping students 
meet the standards, and an emphasis on holding educators and students accountable for student 
achievement—into a single integrated system designed to improve school performance through 
sanctions and incentives is relatively new. 

Standards-based accountability operates through a multilevel, multistep feedback mechanism. 
Content and performance standards that describe what students should know and should be able 
to do establish goals for the education system. Districts and schools are expected to use these 
goals to guide their choices of curriculum, professional development, and other school activities. 
Teachers use the standards as learning goals when they plan instruction. And in this way, the 
coordinated efforts of policymakers, administrators, and teachers promote students’ mastery of 
the desired content and skills. The standards also guide the development of system-wide student 
assessments, and student test scores on these assessments are used as an indicator of school 
success. Incentives are attached to school performance. Schools that do well are rewarded to 
reinforce good practice. Schools that do poorly are offered assistance and, ultimately, sanctioned 
so they will change practice and improve services to students. This feedback loop is intended 
to improve educational practices leading to improved student outcomes. Each state may choose 
different standards, different tests, different assistance policies, and different reward structures, 
but under NCLB, each state must develop some form of standards-based assessment system. 


INCLUDING STUDENTS WITH DISABILITIES IN STATEWIDE 
ASSESSMENTS 


An additional, but often overlooked, stipulation of the [ASA was that each State ensures “par- 
ticipation in such assessments of all students” (IASA, 1994, §1111 (b)(3)(F)(i)). Historically, 
students with disabilities had been exempt from participation in school-level or district-level 
standardized testing requirements. On-grade-level performance was not expected of these stu- 
dents, and exclusion from the large-scale testing was viewed as humane, as well as convenient 
(Crawford, Almond, Tindal, & Hollenbeck, 2002; Thurlow, Seyfarth, Scott, & Ysseldyke, 1997; 
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Ysseldyke & Thurlow, 1994). (Rumors abounded that underachieving students were inflating 
the rolls of special education classes to mask the fact that certain groups of children were not 
learning.) Because the primary target of Elementary and Secondary Education Act regulation 
was the Title 1 program in a school or district, and accountability in federal law (PL 94-142 in 
1975 and the Individuals with Disabilities Education Act [IDEA] of 1990) tended to focus on 
procedural compliance and not on achievement outcomes, little attention was actually paid to the 
IASA ’94 requirement of full participation. 

The situation changed in 1997. The reauthorization of IDEA was a first step in replacing 
the process orientation of the special education compliance monitoring system with a model 
that emphasized academic standards and results (IDEA, 1997). IDEA ’97 again called for full 
participation of students with disabilities in statewide and district-wide assessment programs, 
through the use of reasonable adaptations and accommodations. The amendments asserted that 
the historical underachievement of students with disabilities was linked to low expectations for 
learning and scant access to the general education curriculum (Koenig & Bachman, 2004). In turn, 
mandating that students with disabilities participate in high-stakes accountability assessments 
would promote quality assurance in special education (Defur, 2002). The assumptions were 
that participation raises the stakes, in turn yielding higher expectations, leading to increased 
participation in general education, which promotes better teaching, and results in improved 
academic outcomes for students with disabilities (see Figure 1). This time, the special education 
community took notice and the general education community did as well when the assessment 
and accountability regulations in NCLB (2001) emerged as a logical extension of these IDEA 
’97 provisions. NCLB explicitly prohibited schools from excluding students with disabilities 
from the accountability system. NCLB restated the requirement for participation of all students 
in statewide accountability assessments and reporting of the results for students with disabilities 
with everyone else’s and as a disaggregated group. Furthermore, students with disabilities were 
to be held responsible for the same academic content and performance standards as everyone 
else. 

IDEA 2004, Section 612, Part B, reiterated the NCLB call for including all students with 
disabilities in statewide and district-wide assessment programs. It spoke clearly to the presump- 
tions that tests would adhere to universal design principles to the extent feasible and that most 
students with disabilities would take the regular assessment with appropriate accommodations; 
Section 614, Part B, required Individualized Education Program (IEP) teams to stipulate “why 
a child cannot participate in the regular assessment” and not whether but how he or she would 
(IDEA 2004, §612 (b)). This orientation was consistent with the growing commitment in law and 
public policy to full inclusion of students with disabilities in general education classrooms and 


Participation Higher Increased Better Improved 
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assessment " content 

subjects 





FIGURE 1 Assumptions underlying call for participation of students with disabilities in accountability assessments. 
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access of all students, even those with severe disabilities, to the general education curriculum. 
The orientation also made sense. By excluding no one from the accountability requirement, it 
would be possible to answer truthfully the fundamental accountability question: How well is each 
school doing in bringing all of its students up to standard? The inclusive orientation made the 
educational accomplishments of students with disabilities matter. 


STATES’ RESPONSES TO THE NEW ACCOUNTABILITY 
REQUIREMENTS 


In response to the new mandates, unique statewide assessments were developed or adapted to 
measure how many and which students were attaining targeted knowledge and skills in reading 
and mathematics at certain benchmark grade levels. Further, these state assessments established 
a means of determining school building and school district accountability for each student’s 
attainment of the specified standards. Fifty separate achievement standards were set for what 
constituted proficient academic performance at each of the targeted grade levels in each state 
to answer the accountability question, “By the end of Grade 3 (or 4, 5, 6, 7, 8, 9, 10 or 11) 
what percentage of students in this class (or grade, or school, or district, or state) meet or exceed 
the standard of proficiency?” The answer would be calculated by using the total number of 3rd 
graders (or Sth graders, or 8th graders, or 11th graders, etc.) as the denominator and the number 
who met or exceeded the proficient level on the state accountability assessment as the numerator. 
NCLB set very ambitious goals for schools: By 2014, all students (100%) were expected to meet 
or exceed proficiency levels. Further, NCLB prohibited schools from excluding students with 
disabilities from the accountability system. If the public was to know how well schools were 
educating all their students, they had to include all students in the denominator of calculation. 
That meant including students with disabilities. If those students were not in the denominator, the 
report of what percentage of students were meeting standards would be misleading. Furthermore, 
if they were not in the denominator, all the work of teachers of students with disabilities, and 
all the work of students with disabilities would not count. If these students were not in the 
count, teachers, administrators, school board members, taxpayers, and/or legislators might pay 
less attention to their academic achievement and invest less in improving it. 

By requiring schools and states to aim high, lawmakers believed teachers and students (es- 
pecially special education teachers and students with disabilities) would work harder to achieve 
the “impossible.” As a consequence the academic progress of students with disabilities began 
receiving more attention and scrutiny than ever! 


The 1% Solution 


From the beginning of the requirement for students with disabilities to be included in state- 
and district-wide assessments, lawmakers recognized that there would be students for whom the 
regular assessment, even with accommodations, would not be appropriate. IDEA ’97 stipulated 
that “students with disabilities will be included in state and district-wide assessment programs 
with appropriate accommodations, where necessary, or in an alternate assessment for those who 
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are not able to participate in the general assessment, even with accommodations [italics added]” 
(IDEA 1997, 8612 (a); §300.138). 

NCLB also permitted the development of an alternate assessment based on alternate achieve- 
ment standards for a small population of students for whom the regular assessment, even with 
accommodations, was simply not appropriate. Alternate achievement standards would be aligned 
with a state’s academic content standards, would promote access to the general curriculum, and 
would reflect professional judgment of the highest achievement standards possible (see NCLB 
2001, §200.1(d)). The alternate assessment based on alternate achievement standards would have 
to yield at least three classifications of students (proficient, basic, below basic) so that results 
from that assessment could be aggregated with the results from the regular assessment and all 
students would be included in the accountability reporting. Responsibility for whether students 
with an IEP participated in the accountability system by taking the regular assessment, the regular 
assessment with accommodations, or the alternate assessment based on alternate achievement 
standards was placed in the hands of the IEP team. No cap was placed on the number of students 
with significant cognitive disabilities in a district who could be assigned by the IEP teams to take 
the alternate assessment based on alternate achievement standards, although only 1% of students 
could be counted as proficient based on performance on the alternate. 


Performance of Students With Disabilities Taking the Regular Assessment With 
and Without Accommodations 


The reporting requirements of NCLB and IDEA 2004 require states to include the performance of 
students with disabilities in their annual reports of school, district, and state-level performance and 
to report separately the performance of students with disabilities as a disaggregated group. Over 
the years, participation rates of students with disabilities in the annual accountability assessments 
have increased steadily (see Thurlow, Altman, Cormier, & Moen, 2008), but performance levels 
have remained fairly static (Table 1). In the most recent summary of the performance of students 
with disabilities on state accountability assessments, Thurlow et al. reported that for those states 
for which rates of student proficiency could be calculated for the 2005—06 reading assessment, 


TABLE 1 
Median Rates of Proficient Performance for Students With Disabilities, Nationally, on State Assessments 








Reading Math 
% Increase in Proficiency % Increase in Proficiency 
2002 to 2 
% Proficient in ee eae % Proficient in te OS 
2005-06 No. of States 2005-06 No. of States 
Grade Level Median % Median % Reporting Median % Median % Reporting 
Elementary 44 +6 28 45 +10 28 
Middle 33 +4 27 DH) +4 27 
High School 82 ate) 24 25 aril 24 





Note. Data source: Thurlow, Altman, Cormier, and Moen (2008). 
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slightly more than 30% of students on IEPs performed at a level considered proficient. Proficiency 
rates differed for students at the elementary (44% proficient nationwide), middle school (33%), 
and high school (32%) levels (see Thurlow et al., 2008). Between 2001-02 and 2004—05, median 
change for all states reporting reading scores at the elementary level was an increase of 6 
percentage points across the 4-year span. For the 27 states reporting performance data for middle 
school reading, the median change in percentage proficient across the 4-year period was an 
increase of 4 percentage points. For the 24 states reporting performance data for high school 
reading, the median gain for all states across the 4-year span was 3 percentage points (see 
Thurlow et al., 2008). 

The numbers for mathematics proficiency were virtually the same. For those states for 
which rates of student proficiency could be calculated for the 2005-06 math assessment, gen- 
erally about 45% of elementary-level students, 27% of middle school students, and 25% of 
high school students with IEPs performed at a level considered proficient (see Thurlow et al., 
2008). 

Performance trend data were available for 28 states for elementary math. Between 2001-02 
and 2004—05, the median change across those states was an increase of 10 percentage points. 
Data from 27 states for middle school math showed a median gain across the 4-year span of 4 
percentage points. Data from 24 states for high school math also showed a median increase of 
only | percentage point for the 4-year period (see Thurlow et al., 2008). 

The past several years of Pennsylvania data mirror the national trends. In 2005-06, students 
with disabilities constituted about 15% of the 3rd- through 8th- and 11th-grade population 
of Pennsylvania students participating in the spring statewide accountability assessment, the 
Pennsylvania System of School Assessment (PSSA). Overall, at each grade level, between 60% 
and 75% of students scored at the Proficient level, and about half of the nonproficient students 
performed in the Below Basic category. For students with IEPs, the range of proficiency went 
from a high of 40% at the 3rd grade to a low of 19% at the 11th grade, and about 70% of students 
who were not proficient performed in the Below Basic category (Table 2). 


TABLE 2 
Proficiency of Students on 2005-06 Reading Portion of the Pennsylvania Accountability Assessment, the 
Pennsylvania System of School Assessment 


i 


2005-06 All Students 2005-06 Students With Disabilities 
No. Scored % Proficient+ % Below No.Scored % Proficient+ % Below 
Grade Reading Adv % Basic Basic Reading Adv % Basic Basic 
ee ea a 
3 124,662 69 15 16 17,484 36 18 46 
4 127,680 68 ae 15 19,664 34 29 44 
5 131,488 61 18 21 20,920 24 19 58 
6 135,914 66 18 16 20,913 DS 24 50 
7 141,012 68 17 15 21,170 26 a5 49 
8 143,401 al 13 16 21,370 27 19 54 
11 132,434 65 16 18 16,549 19 19 62 
Total 936,591 67 16 17 138,070 27 21 52) 


a 


Note. Adv = advanced. 
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TABLE 3 
Proficiency of Students on 2006-07 Reading Portion of the Pennsylvania Accountability Assessment, the 


Pennsylvania System of School Assessment 
NS HOG (OC TORS ae ee eee ee eee 





2006-07 All Students 2006-07 Students With Disabilities 
No. Scored % Proficient+ % Below No. Scored % Proficient+ % Below 

Grade Reading Adv % Basic Basic Reading Adv % Basic Basic 

3 125,344 73 12 15 18,041 40 17 43 

4 125,981 70 15 15 19,691 35 20 45 

5 129,593 60 20 20 20,586 23 2) 56 

6 133,399 64 20 17 20,782 23 24 53 

7 138,610 67 17 16 Devan 23 24 53 

8 141,193 75 13 i 21,611 33 OD 44 
11 135,364 65 15 19 17,479 19 17 64 
Total 929,484 68 16 16 139,421 28 a1 51 


i 


Note. Adv = advanced. 


Tables 3 and 4 provide the same data for the two most recent administrations of the PSSA. In 
2006-07, overall proficiency across the state in Grades 3 through 8 and 11 was about 67%, and 
half of the not-proficient students scored at the Below Basic level. Among students with IEPs, 
accounting for almost 15% of the populations of students participating in the spring assessment, 
only 28% achieved at the level of Proficient or above, and more than 70% of students who were 
not proficient performed at the Below Basic level (Table 3). In 2007-08, 31% of students with 
IEPs performed at the Proficient level or above on the reading portion of the PSSA and 70% of the 
students who were not proficient scored in the Below Basic range (Table 4). Proficiency (or lack 
thereof) was not limited to any one of the 13 disability categories eligible for special education 
and related services (Table 5), just as scoring at the Below Basic level was not limited to students 
with disabilities (Figure 2). 


TABLE 4 
Proficiency of Students on 2007-08 Reading Portion of the Pennsylvania Accountability Assessment, the 
Pennsylvania System of School Assessment 


a 








2007-08 All Students 2007-08 Students With Disabilities 
Number Scored % Proficient+ % Below Number Scored % Proficient+ % Below 
Grade Reading Adv % Basic _—_ Basic Reading Adv % Basic _ Basic 
3 126,402 ag) 11 12 19,062 45 17 39 
4 126,266 70 16 14 20,264 36 23 42 
5 127,189 62 20 19 20,506 De 21 a5 
6 130,692 67 18 15 20,589 27 24 49 
if 135,641 70 16 14 21,056 27 25 48 
8 138,339 78 10 12 21,480 37 19 44 
11 134,984 65 16 19 18,101 19 19 62 
Total 919,513 70 15 15 141,058 31 21 48 





Note. Adv = advanced. 


TABLE 5 
Percentage of Students by Disability Category not Proficient on the 2005-06 Reading Portion of the 
Pennsylvania Accountability Assessment, the Pennsylvania System of School Assessment (PSSA) 
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% not Proficient in 2006 PSSA Reading 


Disability Category 





Traumatic brain injury 69 
Hearing impairment including deafness 63 
Specific learning disability 77 
Mental retardation 60 
Orthopedic impairment SI 
Emotional disturbance 77 
Speech or language impairment 37 
Visual impairment including blindness 54 
Deaf/Blind 59 
Multiple disabilities 719 
Autism 56 
Other health impairment 65 
Total 64 
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Legitimate questions have been raised regarding the low proficiency rates of students with 
disabilities on the statewide assessments. Which facet of the law’s quality assurance equation 
(depicted in Figure 1) is erroneous? Were students not exposed to the general education curriculum 
covered on these assessments? Would substantially more students with disabilities have learned 
to standard if they had had proper access to the curriculum? Were students with disabilities being 
taught by teachers who were not highly qualified? Would more students with IEPs have been able 
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to perform at proficient level if taught by different teachers? Were students actually performing 
at proficient levels but their ability was misrepresented because the test was not constructed 
using appropriate Universal Design concepts and student performance was laden with construct 
irrelevant variance? Or were none of these the case? Maybe students with disabilities had been 
exposed to the appropriate grade-level curriculum, had been taught by highly qualified teachers, 
and had learned all they could in the time available—they just could not learn everything that 
needed to be learned within 1 school year to achieve proficiency. 


The 2% Solution 


Given the growing evidence of large numbers of students with disabilities unable to meet pro- 
ficiency standards in every state across the nation, and the serious consequences for students 
in disaggregated groups not achieving the targets for Adequate Yearly Progress (AYP), there 
developed a groundswell of public concern for the “gap kids.” These students are not sufficiently 
cognitively impaired to qualify for an alternate assessment based on alternate achievement stan- 
dards yet are not learning enough to reach proficiency on the standard assessment even with 
accommodations. In December 2005, the U.S. Department of Education responded to these pres- 
sures and concerns from the field and announced greater flexibility in the participation of students 
with disabilities in the statewide assessment and accountability systems. The flexibility would 
apply to “a group of students with disabilities who can make significant progress, but may not 
reach grade-level achievement standards within the same time frame as other students, even after 
receiving the best-designed instructional interventions from highly trained teachers” (Federal 
Register, Vol. 70, 2005, p. 74624). Similar to the flexibility afforded to states and school districts 
for students with the most significant cognitive disabilities (i.e., the 1% students), the proposed 
change would allow a state to use a “documented and validated standards-setting process to de- 
fine modified [italics added] achievement standards for some students with disabilities” (Federal 
Register Vol. 70, 2005, p. 74625). Schools would still be required to assure that all students had 
access to grade-level curriculum, that instruction was aligned with the state’s academic content 
standards for the grade in which the student is enrolled, and that assignment to the new alternate 
assessment would not preclude a student from earning a regular high-school diploma (Federal 
Register, Vol. 70, 2005, p. 74625). Students eligible to take assessments based on modified 
achievement standards could come from any of the 13 disability categories listed in the IDEA, 
and IEP teams would review on an annual basis their decision to assign a student to an assessment 
based on modified achievement standards to ensure that the decision remains appropriate (Federal 
Register, Vol. 70, 2005, p. 74625). 

The December 2005 change in policy was intended to address the problems of the “gap” 
students, students for whom the “grade level assessment is too difficult, and therefore does 
not provide data about the student’s abilities or information that would be helpful in guiding 
instruction” and for whom the 


alternate assessment based on alternate achievement standards is too easy, . . . not intended to assess a 
student’s achievement across the full range of grade level content, . . . [and] would not provide teachers 


and parents with information to help these students progress towards grade level achievement. (U.S. 
Department of Education, 2007, p. 8) 


THE “TWO PERCENT STUDENTS” 487 


But the announcement left it to the individual states to define the target population within 
specified parameters. First, the new flexibility should help students with persistent academic 
disabilities and served under IDEA who, because of their disability, may not be able to achieve 
grade-level proficiency within the same time frame as other students, even after receiving the best- 
designed instructional interventions, including special education and related services, designed 
to address the student’s individual needs from highly trained teachers. This parameter reflected 
the overwhelming federal commitment to assessing all students on grade-level content and to 
ensuring that all students are working toward mastering grade-level content. Second, the proposed 
flexibility would require that modified achievement standards be aligned with grade-level content 
but reflect reduced breadth or depth of that grade-level content so that a small group of students 
with disabilities would be better able to demonstrate what they know and can do on a state test. 
Third, the new provision would not limit how many students could be assessed against modified 
achievement standards and left the decision about which students should take such an assessment 
in the hands of the IEP team. Furthermore, states could include proficient scores from such 
assessments in making AYP decisions, with the proviso that such scores be capped at 2% of 
the total tested population.' Like the regulation for students with the most significant cognitive 
disabilities, the new flexibility was tailored to the needs of particular students, while ensuring the 
goals of NCLB were still achieved. 

In justifying the flexibility, Secretary Spellings’s press release proclaimed that “the latest 
scientific research shows that students with disabilities ... can make progress toward grade-level 
standards when they receive high-quality instruction and are assessed with alternate assessments” 
(U.S. Department of Education, 2005a). An accompanying fact sheet (Ed.gov, 2005) advised 
that “about 2% of children ...are not able to reach grade-level standards, even with the best 
instruction” and refer the reader to 


a review by [Lyon, Fletcher, Fuchs, & Chabra, 2007] indicating that “the best-designed instructional 
interventions achieved a range of success from a low of 50% to a high of 90% of participating 
students reaching grade-level reading standards. Those students who did not respond well to these 
interventions (approximately 10-50%) are .. .0.5% to 3% of the total population.” 


The same fact sheet (U.S. Department of Education, 2005b) went on to cite the following 
research to justify the 2% figure: A study by Torgeson et al. (1999) in which 


most of the struggling students who received explicit reading instruction attained average levels of 
reading achievement, but 24% of these students did not reach grade level standards. Extrapolated to 
the population at large, that 24% of low responders reflects about 2.4% of the total student population 
who were unable to attain grade-level reading achievement (Ed.gov, 2005). 


1<Under specific limited conditions, States and {local educational agencies, or] LEAs, may exceed the 2.0% cap. The 
2.0 percent cap may be exceeded only if a State or LEA is below the 1.0 percent cap for students with the most significant 
cognitive disabilities who take alternate assessments based on alternate academic achievement standards. For example, 
if the number of proficient and advanced scores on the alternate assessment based on alternate academic achievement 
standards is 0.8 percent, the State or LEA could include 2.2 percent of the proficient and advanced scores on alternate 
assessments based on modified academic achievement standards in calculating AYP” (U.S. Department of Education, 
2007, p. 34). 
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And layered intervention studies by Mathes et al. (2005) and McMaster, Fuchs, Fuchs, and 
Compton (2005) in which the intervention did “not work adequately with 10% of the bottom 20% 
of the population yield[ing] an inadequate response rate of 2%” (Ed.gov, 2005). Thus, the fact sheet 
concluded, 2% of students assessed in the annual accountability assessment, or approximately 
20% of students with disabilities, was a reasonable and sufficient cap for the new flexibility. 

The same press release (U.S. Department of Education, 2005a) and fact sheet (Ed.gov, 2005) 
referenced (and then ignored) remedial reading research studies of students with reading diffi- 
culties in which nearly 80% of the research samples failed to achieve at grade level at the end of 
the treatment; that extrapolates to 16% of the school population, not 2% of the school population 
(Foorman et al., 1997; Klingner, Vaughn, Hughes, Schumm, & Elbaum, 1998). Nevertheless, the 
new policy was promoted to allow those students with persistent academic disabilities to take 
an assessment that is sensitive to measuring progress in their learning and that recognizes their 
individual needs, although they capped the number of scores that could be counted as proficient 
based on modified achievement standards at 2%. 

With this limited guidance, states were permitted to develop modified achievement standards 
and provide alternate assessments for their target group of students with disabilities. It was left 
to the individual states to define more clearly who the target population was, and to develop 
guidelines to aid IEP teams in making the decision of which students with disabilities would take 
which version of the annual statewide accountability assessment—the regular assessment with 
or without accommodation, the alternate assessment with performance judged against modified 
grade-level achievement standards, or the alternate assessment with performance judged against 
alternate grade-level achievement standards. 


WHO ARE (OR SHOULD BE) THE 2% STUDENTS? 


The U.S. Department of Education spokespersons admitted that IEP teams would likely find it 
“more difficult ...to identify students eligible for an alternate assessment based on modified 
achievement standards than .. . for an alternate assessment based on alternate achievement stan- 
dards” (Federal Register, Vol. 70, 2005, p. 74626). They suggested that students in the “2% group” 
would not necessarily be the lowest achieving 2% of students, nor would the “2% students” be 
all students with IEPs who are having difficulty with grade-level content or who are receiving 
instruction below grade level. They challenged each state to design criteria to help IEP teams 


distinguish between students whose disability has truly precluded them from achieving grade-level 
proficiency and those who, with appropriate services and interventions, including special education 
and related services designed to address the student’s individual needs, can be assessed based on 
grade-level achievement standards. (Federal Register, Vol. 70, 2005, p. 74626) 


To assist in the discussion of the choices that need to be made, Figure 3 is presented as a graphic 
representation of the distribution of scores of students with disabilities on statewide assessments. 
Using the 2008 Pennsylvania figures presented in Table 4, and rounded off for simplicity, Figure 
3 shows 30% of students with disabilities scoring Proficient (25%) or Advanced (5%), and 70% 
of students with disabilities scoring below proficiency (students A—-N), some nearly proficient 
(represented by students C and A) and some far below proficiency (represented by students J, L, 
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FIGURE 3 Hypothetical distribution of students with disabilities on statewide assessment. 


and N). In this graphic, each circle represents 5% of students with disabilities taking the regular 
grade-level statewide accountability assessment, with or without appropriate accommodations. 

The special education students who are likely to be identified to take an alternate assessment 
based on modified achievement standards may come from any of the 13 disability categories 
defined in IDEA (1990)(§200.1(f)(1)(11)). “The students who participate in assessments under 
this option are not limited to those who are close to achieving grade level (represented by student 
A in the graphic above) or who are relatively far from achieving at grade level (represented by 
student N in the graphic above)” [(material in italics added] (Federal Register Vol. 72, 2007, p. 
17749). 


Choosing the “Almost Proficient” 


Much of what has been written about the target population for this assessment option (see Lazarus, 
Thurlow, Christensen, & Cormier, 2007; Quenemoen, 2008) makes it appear as if the new alternate 
assessment based on modified achievement standards is intended for the students represented by 
Circle A. These are students whose progress to date, in response to appropriate instruction 
(including special education and related services) in the grade-level curriculum, shows that they 
have learned some grade-level content but have been unable to achieve grade-level proficiency 
within the year covered by their IEP (described in Sec.200.1(e)(2)(ii)(A)). One school year would 
appear to be not enough time in which to master grade-level content. For example, these students 
might need more than 1 year in which to learn the fourth-grade math curriculum, but given some 
extra time, they might actually demonstrate proficiency in fourth-grade mathematics. Currently 
at the end of fourth grade, these students must take the statewide accountability assessment 
covering all of fourth-grade content, and although they have mastered some of it, they cannot 
perform well enough to earn a proficiency rating. The next year, they move on to the fifth-grade 
curriculum and take the fifth-grade accountability assessment. Again, they are highly unlikely to 
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score at a proficient level on the fifth-grade test because they would need to learn all of the rest 
of fourth-grade curriculum and all of fifth-grade curriculum in that fifth-grade year and they are 
unlikely to do so. 

Hypothetically, one could imagine giving these fourth-grade students longer than | year in 
which to become proficient on fourth-grade level standards. In fact, by high school, these are the 
very struggling students for whom districts design elongated English, math, or science courses. An 
algebra course that is usually completed in 1 school year is elongated into a 2-year course so that 
students have more time for explicit instruction and practice before taking the end-of-course exam 
(see Aguilar, Morocco, Parker, & Zigmond, 2006; Brigham, Parker, Morocco, & Zigmond, 2006; 
Morocco, Clay, Parker, & Zigmond, 2006). In our fourth-grade example just given, at the end of 
the 1st year in fourth grade, the students might take an alternate fourth-grade assessment based on 
modified fourth-grade level achievement standards so that the substantial progress they have made 
in the fourth-grade level curriculum could be acknowledged and counted. But instead of moving 
on to fifth grade, these students would continue learning the fourth-grade level academic content, 
then take the regular fourth-grade level assessment at the end of the year. In this second year, 
proficiency would be judged against the regular fourth-grade performance level descriptions. This 
could be a sensible idea. After all, in a standards-based educational environment the standards 
are supposed to be fixed, but the students given a longer or shorter time to learn what they 
need to achieve proficiency (Educational Broadcasting Corporation, 2004). In a standards-based 
system, students usually can’t progress past some benchmark points (often at Grades 4, 8, and 
12) without demonstrating the knowledge and skills written in the standards documents. But in 
practice, students with persistent academic problems could need to spend 2 years in each grade 
to have enough time to master the general education curriculum and “pass” the state test. That 
turns this into an absurd idea, expensive not only in students’ time (adding at least 5 years to 
K-12 schooling) but also in terms of a district’s dollar costs (increasing grade level enrollments 
without increasing the tax base). 

If the alternate assessment based on modified achievement standards is aimed at the students 
represented by Circle A in Figure 1, and those students don’t spend 2 years in each grade, it is hard 
to imagine why the IEP team would be required to make the participation decision annually. If a 
student cannot learn enough in 1 year to demonstrate proficiency on the fourth-grade academic 
content standards, it is hard to imagine how in the next academic year, he or she could learn 
the rest of the fourth-grade curriculum and all of the fifth-grade curriculum. Surely at the end 
of the fifth grade, this student would again take the assessment based on modified achievement 
standards. In other words, once placed in this new assessment option, the IEP team would be 
unlikely to return the student to the standard assessment. This could lead to a kind of “educational 
tracking” for the students represented by Circle A, an undesirable outcome on its own (Oakes, 
Wells, Jones, & Datnow, 1997; Wheelock, 1992), but also in direct conflict with the goals of 
inclusion—in the traditional sense of students sitting with peers. 

Of course, students with disabilities are not the only ones scoring below proficiency on the 
state assessment (see Tables 2, 3, & 4). But the 2% option is not available for students without 
IEPs, so the slower track, less demanding curriculum could be available to other students but 
not the possibility of being called proficient based on modified achievement standards. Also, it 
is hard to imagine how one could target the students represented by Circle A and not extend the 
option to all the students with IEPs who scored even lower than those students (represented by 
Circles B-N). It would be difficult to explain to an JEP team, or to the parents of a student with 
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disabilities struggling with academic achievement, that students who have almost mastered the 
general education curriculum would be allowed to take an “easier” test (easier in the sense that the 
student does not have to know quite as much to be judged proficient on the subject matter) while 
students who are unable to master the general curriculum at all must still take the “harder” version 
of the statewide assessment. Because there is no cap on the number of students who may take the 
alternate assessment based on modified achievement standards, it is likely that as many as 60% 
to 70% of students with disabilities (with proportions of students likely increasing as grade levels 
increase) would be assigned to the new 2% assessment. This would make “tracking” an even 
more likely and an undesirable outcome of this new option. The 2% assessment would become 
the “special education assessment” instead and all the work in fashioning higher expectations for 
students with disabilities would be undermined. 

Finally, we have seen significant improvement in the academic attainment of students with 
disabilities in the last decade. The feedback loop appears to be working. As more and more 
students with disabilities participate in the statewide assessment each year and everyone works 
harder and harder to help them reach grade-level standards, more and more students experience 
academic growth (Table 1). That means that the students who are currently almost proficient 
(Circle A) might soon become proficient with even greater effort, improved instruction, and 
explicit intervention. Assigning these students to the modified assessment would allow teachers 
to give up on the goal of reaching those higher expectations, rather than to try harder to teach 
them—a situation not to be encouraged. 


Choosing the “Nowhere Near Proficient” 


The alternative is to target the new assessment flexibility to students represented by Circle N in 
Figure 3. These are students whose current scores on the statewide assessment are the farthest 
from being proficient. These are students with significant difficulties in learning the general 
education curriculum at their grade level. Before this new assessment was made available, these 
students could be assigned to take either the grade-level assessment or an alternate assessment 
based on alternate achievement standards. Neither option provides an accurate assessment of 
what these students know and can do. Academic achievement growth trajectories suggest that 
students represented by Circle N are extremely unlikely to become proficient in subsequent years. 
These are students who can learn some grade-level content, information, and facts but who are 
very poor readers and/or very poor at math. These are students who might be able to participate 
in science, social studies, and literature discussions but who cannot read the textbooks or trade 
books, cannot cover as much content as other students, cannot cover the content to the same 
depth, and cannot learn the content to the same degree of cognitive complexity. 

Aiming an alternate assessment based on modified grade-level achievement standards in 
reading and/or mathematics at this group (represented in Figure 3 by Circle N) avoids some of 
the undesirable consequences of targeting their higher performing peers (Circle A) discussed 
earlier. Focusing on the lowest performing students directs IEP teams to consider a smaller 
number of students for the new option. It changes academic achievement expectations only for 
a small group of students with disabilities instead of creating a special education “track” for all 
nonproficient students with disabilities. It gives these very, very hard to teach students credit for 
the extraordinary effort it will take to achieve any grade-level competence. 
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But several problems remain. First, can we reasonably and responsibly call their achievements 
“proficient on grade level content” when students have mastered fewer skills and less sophisticated 
concepts in a narrowed curriculum? Is it fair to the students represented by Circle A, their 
parents, and their teachers to change the definition of proficiency and “pretend” the students 
represented by Circle N are proficient on grade level work? Who benefits from this charade? 
Second, what will be the cumulative effects of redefining proficiency standards grade after grade? 
If the modified achievement standards say that third-grade students need master only some 
multiplication concepts but not others, think about the implications on successfully applying 
division concepts in fifth-grade or algebraic concepts in eighth grade and so on. Third, how will 
teachers and schools balance their commitment to inclusion with a classroom reality in which 
some students are not required to learn what the teacher is teaching or the teacher is moving too 
quickly through material they need to master? Developing modified standards for some students 
will require eliminating some skills and extending the coverage time for others because the 
students cannot learn the entire curriculum in | year’s time. How will the 2% students and their 
parents, their classmates and their classmates’ parents, the remaining students with disabilities 
struggling on the regular statewide assessment and their parents, as well as the general public 
react when Johnny, a student represented by Circle N, is called “proficient” based on modified 
achievement standards, whereas his classmate George, represented by any of the Circles A through 
M, is called “not proficient” on the statewide test? 


CONCLUDING COMMENTS 


The new assessment flexibility assumes the appropriateness of allowing an additional small 
percentage of students with disabilities (beyond students with significant cognitive disabilities) 
to be assessed based on modified academic achievement standards aligned with the state’s grade- 
level academic content standards. It allows the performance of this small percentage of students to 
be judged as “proficient” even if the breadth or depth of those grade-level standards is reduced or 
the format or design of the grade-level assessment is different. It also allows the target population 
for this flexibility to be defined at the state level, by bureaucrats and testing companies instead of 
special education leaders and researchers. The nonregulatory guidance to IEP teams is confusing. 
The regulations (§200.1(e)(2)(Gi)(A)) already stipulate that a student should not be assigned to an 
alternate assessment based on modified achievement standards if the child’s IEP is not of “high 
quality” and “specially designed to . .. move [the] child closer to grade-level achievement, if the 
child is not already achieving at grade level.” Does that mean a struggling student with a poorly 
written IEP will be precluded from taking the new alternate assessment for at least a year during 
which time the IEP team writes a better IEP? 

Further, Section 200. 1(e)(2)(ii)(A) stipulates that students should not be assigned to an alternate 
assessment based on modified achievement standards if they have not had the opportunity to 
learn grade-level content. Does that mean that a struggling student being taught reading or 
mathematics at his/her instructional level rather than at “grade-level” would be precluded from 
taking the new alternate assessment for at least a year during which time he is provided less 
appropriate and ‘special’ grade level instruction instead? Section 200.1(e)(2)(ii)(B) requires IEP 
teams to examine a student’s progress in response to high-quality instruction over time and using 
multiple measures before assigning that student to an alternate assessment based on modified 
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achievement standards. Does that mean that a struggling student whose teacher delivers “less than 
high-quality instruction” would be precluded from assignment to the new assessment until she 
or he gets a better teacher? And how should measuring student progress be standardized? Given 
the pressures of accountability and the time and effort devoted to preparing for and administering 
state tests, many schools have stopped administering additional standardized achievement tests. 
And, although Response to Intervention and Reading First have promoted increased use of 
curriculum-based-measurement and progress monitoring in reading and math, these models are 
often restricted to the elementary level. In fact, the field is only beginning to scratch the surface 
in developing and implementing comprehensive progress monitoring/intervention frameworks 
for middle and secondary students—the primary cohort for accountability testing (Mastropieri 
& Scruggs, 2005; Vaughn et al., 2008). How will IEP teams determine how much progress is 
sufficient? And for how long should an IEP team wait for signs of progress before recommending 
the student for the new alternate? 

It is too soon in the implementation of the new flexibility to paint a broad picture of how states 
are responding to the challenge of defining who the 2% students are. To date only five states 
have undergone peer review of their documentation related to this new assessment option (see 
Filbin, 2008). The National Center on Educational Outcomes (Lazarus et. al, 2007) summarized 
the eligibility criteria used by these states to define which students with IEPs would participate 
in the new test. All states required that students have an IEP to be eligible. Exclusionary criteria 
paralleled those detailed in the federal regulations (i.e., participation is not based on a disability 
label, students do not have a significant cognitive disability, etc.). Inclusionary criteria for four 
states required that students’ performance be multiple years behind grade-level expectations. In 
fact, one state required that students function at least 3 years below grade level. Three states re- 
quired that students’ instruction be significantly changed in scope, breadth, or complexity relative 
to the grade-level standards. Four states required that students have had intensive individualized 
instruction and support, whereas two specifically state that students must have received at least 3 
years of intensive individualized instruction in reading and/or math. Even such a brief snapshot 
of these states’ efforts to define the 2% population underscores the aforementioned considera- 
tions about possible unintended consequences of defining participation on inclusive education, 
curriculum design, cumulative learning, and long-term school experience. 

It is also too soon to determine whether clear enough guidelines can be developed for IEP teams 
to help them make the subtle distinctions between students eligible for the alternate assessment 
based on alternate achievement standards, alternate assessment based on modified achievement 
standards, and regular assessment with appropriate accommodation. It is too soon to know which 
states will go in which direction and what impact this diverse implementation will have on the 
special education field. It is too soon to know how many students will be assigned to this option. If 
only a small percentage of students are assigned to the new assessment option, how will schools 
address the underachievement of the majority of students with IEPs not eligible to be called 
proficient by either alternate assessment option? And if a large number of students are assigned to 
the new assessment option, how will schools and IEP teams respond when the percent of students 
with disabilities reported as proficient stabilizes because the 2% quota is exceeded? 

During the past 2 decades, considerable effort has been exerted at both federal and state 
levels on reforms designed to ensure that all of America’s students are prepared for work in an 
increasingly competitive global society. The emphasis nationwide has been on improving results 
of educational efforts by specifying desired achievements for students and developing systems 
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of accountability to evaluate the attainment of those desired achievements. We have seen many 
positive outcomes from the requirement to include students with disabilities in this standards- 
based reform movement and in the assessment and accountability provisions of NCLB. The 
policies related to students with disabilities in NCLB have made more visible and concrete our 
nation’s commitment to the inclusion of students with disabilities in the mainstream of education, 
to their exposure to and mastery of the general education curriculum, and to the desirability 
of setting higher and higher expectations for their educational attainments. The jury is still out, 
however, on whether this new assessment flexibility and the uncertainty about who itis for will help 
or hinder progress of students with disabilities in attaining high levels of academic achievement. 
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Several states had an assessment that they considered to be an alternate assessment based on modified 
academic achievement standards (AA-MAS) in place, or in development, when the April 2007 federal 
regulations on modified achievement standards were finalized. This article uses publicly available 
information collected by the National Center on Educational Outcomes to analyze changes in states’ 
AA-MAS between 2007 and 2008. The article compares across the 2 years the number of states 
that had an assessment they considered to be an AA-MAS, states’ participation guidelines, and the 
characteristics of these assessments. We also provide information about the number of students who 
participated in this assessment option during the 2006-07 school year. In 2007, six states (Kansas, 
Louisiana, Maryland, North Carolina, North Dakota, Oklahoma) either had or were in the process of 
developing an assessment they considered to be an AA-MAS. In 2008, California, Connecticut, and 
Texas also offered this assessment option. Six of the nine states that have been early implementers 
of the AA-MAS were states that had offered an out-of-level testing option until federal policies 
required that option to be phased out. Most states had fewer items on their AA-MAS than on their 
regular assessment. States that had participation guidelines for the AA-MAS in 2007 developed them 
prior to finalization of the federal regulations, and some changes between 2007 and 2008 in states’ 
participation guidelines appear to have been made to better align the guidelines with the regulations. 
The characteristics of the assessments states consider to be AA-MASs are changing rapidly and will 
probably continue to change rapidly over the next few years as more is learned about the advantages 
and limitations of this assessment. 


Federal legislation requires that all students, including students with disabilities, be included in 
state accountability systems. Most students with disabilities participate in the regular assessment 
with or without accommodations, and a few with the most significant cognitive disabilities par- 
ticipate in alternate assessments based on alternate achievement standards. A couple states also 
have an optional assessment called an alternate assessment based on grade-level achievement 
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standards for those students with disabilities who need alternate ways to show what they 
know. ! 

In April 2007 No Child Left Behind regulations gave states the flexibility to offer another 
assessment option, an alternate assessment based on modified academic achievement standards 
(AA-MAS), for students with disabilities. The AA-MAS is an assessment option for students with 
an Individualized Education Program (IEP) whose progress to date, in response to appropriate 
instruction, is such that the student is unlikely to achieve grade-level proficiency within the school 
year covered by his or her IEP. The AA-MAS is sometimes referred to as the “2% option” because 
states may count up to 2% of all students as proficient based on the percentage of students who 
receive special education services and who meet proficiency standards with this option (Lazarus, 
Thurlow, Christensen, & Cormier, 2007). 

Large-scale assessments developed specifically for students with disabilities are a part of states’ 
assessment and accountability systems. This means that they must conform to the same criteria 
that guide the development and administration of other assessments (e.g., American Educational 
Research Association, American Psychological Association, National Council on Measurement 
in Education, 1999). 

The purpose of this study was to examine the evolution of AA-MAS. Specifically, it was 
designed to examine changes that have occurred in states’ approaches to, and student participation 
in, the option since the finalization of the regulations in 2007. Specifically, our research questions 
were as follows: 


Did the number of states that offered an AA-MAS change between 2007 and 2008? 
Have states’ eligibility criteria for the AA-MAS changed substantially since 2007? 

Has the percentage of students participating in the AA-MAS changed across time? 

Has the test design of states’ AA-MAS changed substantially since 2007? 


2 Ss 


BACKGROUND INFORMATION 


In 2006-07, all states had both general assessments and alternate assessments based on alternate 
achievement standards; in addition, a couple states (e.g., Massachusetts, North Carolina) had 
alternate assessments based on grade-level achievement standards (Albus, Thurlow, & Bremer, 
2009). A few states (e.g., Kansas, Louisiana, North Carolina, North Dakota, Oklahoma) also had 
an assessment they considered to be an AA-MAS @Lazarus et al., 2007), having gone ahead and 
developed an assessment in response to the 2005 Notice of Proposed Rulemaking about the AA- 
MAS (Notice of Proposed Rule Making: Title I, 2005). At least one additional state (Maryland) 
had an AA-MAS under development prior to the April 2007 release of the regulations (Lazarus 
et al., 2007). As of November 2008, none of the states that believed they had an AA-MAS had 
successfully completed the federal peer review process used by the U.S. Department of Education 


‘Ror a detailed comparison of alternate assessment options (e.g., AA-AAS, AA-GLAS, and AA-MAS), refer to 
the table in Appendix A, “Modified Academic Achievement Standards: Nonregulatory Guidance” (U.S. Department of 
Education, 2007, pp. 52-53). 
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to determine whether the assessment satisfies the requirements for the state to receive federal 
funds.” 

According to Koretz (2008), the fundamental dilemma, which was highlighted by the regula- 
tions, is “the difficulty of deciding how best to test low-achieving students” (p. 309). Almond, 
Quenemoen, Olsen, and Thurlow (2000) explored issues surrounding students who did not neatly 
fit into the assessment options that were available at the time their study was conducted. They 
concluded that these issues reflected “gray areas of assessment” and that the challenge of students 
who do not fit into existing assessment systems is not really due to “gray area students,” but rather 
to issues with the assessment system itself (i.e., “gray areas of assessment systems’). It was sug- 
gested at that time that the gray areas were not the same everywhere and that the specific issues 
that existed were related to the context in which they emerged. States developing an AA-MAS 
are doing so within the context of their existing assessment systems. 


Participation 


Prior to the development of the regulations that permitted the AA-MAS option, some educa- 
tors, advocates, and policymakers asserted that there may be some students with disabilities 
whose knowledge and skills were not well assessed with assessment options available at that 
time. However, the data suggest that only a small proportion of the population of students with 
disabilities, overall, has not participated in state assessments. The combined rates of partici- 
pation of students with disabilities across the assessment options were quite high (Thurlow, 
Altman, & Cormier, 2009), ranging from 95.3% of students with disabilities on both high 
school reading assessments and high school mathematics assessments to 98.8% of students 
with disabilities on both elementary reading assessments and elementary mathematics assess- 
ments. Most states reached these levels of participation without the AA-MAS. There is still 
some variation across states, but in general participation of students with disabilities 1s com- 
parable with other student subgroups. For example, Thurlow, Bremer, and Albus (2008), in 
an analysis of publicly reported participation data on the eighth-grade general math assess- 
ment, found that in 10 of the 14 states 97-99% of students with disabilities participated in the 
assessment. 

Every state has guidelines for participation for their various assessments, and they formulate 
each set of guidelines to be consistent with the purpose and objectives of their large-scale as- 
sessments. Analyses of the 2007 participation guidelines of all 50 states (Christensen, Lazarus, 
Crone, & Thurlow, 2008) showed that states most often recommended that the following criteria 
be used to determine the way in which students participate in their assessments: IEP determina- 
tion (50 states), instructional relevance/ instructional goals (44 states), current performance/level 
of functioning (36 states), and level of independence (36 states). States also indicated criteria 
that should not be considered: presence or category of disability (25 states); cultural, social, 
linguistic, or environmental factors (23 states); and excessive absences (22 states). The partici- 
pation guidelines of states that offer an AA-MAS also include criteria for participation in that 
option. 





2 As of a date (July 2009) near the time of publication one state (Texas) had received approval. 
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Performance 


Although participation rates in the existing assessments were quite high, the same was not true, in 
general, of the performance of students participating in the assessments. In 2006-07, an average of 
38.9% of students with disabilities showed proficient or above performance on the general reading 
assessment at the elementary school level, and an average of 40.6% of students with disabilities 
showed proficient or above performance on the general mathematics assessment at the elementary 
school level (Thurlow et al., 2009). This compares with 62.4% of all students showing proficiency 
on the elementary reading assessment and 67.7% on the elementary mathematics assessment.? It 
is expected that the proficiency rates of students with disabilities will conform to the trends of 
increasing performance shown by students with disabilities across time (Thurlow, Quenemoen, 
Altman, & Cuthbert, 2008). 


Context of Current Study 


It is important to examine any new assessment that is introduced into an existing assessment 
system. Just as we are interested in whether states are determining that they need another option 
(specifically, the AA-MAS), and whether the need for this assessment has changed across time, 
we are also interested in who any new assessment is for and what the test does differently from 
what other assessments do. In a particularly active period for assessments, where there is increased 
attention to the effects of the performance of subgroups on accountability results (Koretz, 2008; 
Meier & Wood, 2004), it is essential that we watch and document the evolution of assessments. 


METHOD 


To obtain the information for the current analysis, data were gathered from state Web sites and 
other publicly available sources. Data collected included (a) states’ participation guidelines for the 
AA-MAS, (b) participation data for states with an AA-MAS, and (c) information about selected 
test design changes for states’ AA-MAS. The participation guidelines and information about the 
assessment design characteristics were downloaded from states’ Web sites during summer 2007 
and summer 2008. The states included in this study had an assessment the state considered to be 
an AA-MAS in place in summer 2007 or summer 2008—or had information about an AA-MAS 
in development on the state’s Web site. 

The National Center on Educational Outcomes (NCEO) has tracked characteristics of states’ 
AA-MAS since 2007 (Albus, Lazarus, Thurlow, & Cormier, 2009; Lazarus, Rogers, Cormier, 


3The percentage of all students who show proficiency on the elementary reading and mathematics assessments was 
estimated using data in Mapping 2005 State Proficiency Standards Onto the NAEP Scales: Research and Development 
Report (National Center for Education Statistics, 2007). For fourth-grade reading and mathematics, the average state 
proficiency rate was calculated by summing the “estimate of proportion meeting the state proficiency standard” for all 
states included in the report, and then dividing by the number of states. It was assumed that the proportion proficient in 
fourth grade was similar to the proportion proficient in other tested elementary grades. 
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& Thurlow, 2008; Lazarus et al., 2007). Participation data also were collected from states’ Web 
sites, starting with those states that had an AA-MAS in 2008. 

States were given the opportunity to verify that the participation criteria and test design 
information that we gathered was correct. If a state requested a change, NCEO required written 
documentation as to the source of the requested change before making the change. Data for states’ 
participation guidelines and selected characteristics of the states’ AA-MAS were collected, 
organized, and reported by NCEO in 2007 and 2008. See Albus et al. (2009), Lazarus et al. 
(2008), and Lazarus et al. (2007) for a complete list of documents used, and for additional 
details and specifications. If the NCEO reports did not contain detailed information about a 
specific characteristic or criterion, relevant documents from the states were reviewed to gather 
additional data. For the participation criteria and the test design changes, we compared changes 
between 2007 and 2008 in the number and percentage of states that included specific criteria 
or characteristics. For the participation data, we examined those numbers or rates provided for 
students with disabilities in the AA-MAS. 


RESULTS 


The Results section contains five subsections. First we report on which states had an AA-MAS 
in 2008 and then compare this with the states that offered this option in 2007. The second 
section contains a detailed comparison of states’ participation guidelines for the AA-MAS in 
2007 and 2008. Next we report the participation rates in states with an AA-MAS. Following this 
are the results of our analysis of the AA-MAS assessment and question characteristics for states 
that offered this option in 2007 and 2008. The final subsection analyzes how states’ AA-MAS 
differed from their regular assessment in 2007 and 2008. As described in the Methods section of 
this article, the reported results for 2007 and 2008 included both the states that had an assessment 
the state considered to be an AA-MAS and the states that had information about an AA-MAS in 
development on their Web sites. 


States With an AA-MAS 


The number of states that believed they had an AA-MAS increased from six states (Kansas, 
Louisiana, Maryland, North Carolina, North Dakota, Oklahoma) in 2007 to nine in 2008 
(California, Connecticut, Kansas, Louisiana, Maryland, North Carolina, North Dakota,Oklahoma, 
Texas). As previously noted, as of November 2008,* no states had successfully completed the 
U.S. Department of Education’s peer review process for the AA-MAS. Table 1 shows the names 
of the AA-MAS in these states, along with the content areas and grades for which the assessments 
were developed. All nine states had an AA-MAS for mathematics and reading/English language 
arts. California, Kansas, Louisiana, North Carolina, North Dakota, Oklahoma, and Texas also 
had an AA-MAS for science. Kansas, Louisiana, and Texas had a social studies AA-MAS. Some 
states offered the AA-MAS at more grade levels than others. 


4See footnote 2. 
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TABLE 1 
Alternate Assessment Based on Modified Academic Achievement Standards Name, Content Areas, and 
Grade by State, 2008 








State Assessment Name Content Areas/Grades 
California California Modified Assessment ELA (3-8); Math (3-7); Science (5,8) 
Connecticut CMT/CAPT Modified Achievement Standards ELA and Math (3-8,10-11) 

Kansas Kansas Assessment of Multiple Measures Reading and Math (3-8; once in HS), Writing 


(5,8, once in HS); History/Government (6,8, 
once in HS); Science (4,7, once in HS) 


Louisiana LEAP Alternate Assessment, Level 2 English and Math (Grades 4-10); Science and 
Social Studies (4, 8, 11) 
Maryland? Modified Maryland School Assessment and Reading/ELA and Math (3-8, HS) 
Modified High School Assessment 
North Carolina NCEXTEND2 Reading and Math (3-8); Science (4,8,11) 


North Dakota North Dakota Alternate Assessment Aligned to Reading and Math (3-8); Science (4,8,11) 
North Dakota Content Standards for Students 
with Persistent Cognitive Disabilities 


Oklahoma Oklahoma Modified Alternate Assessment ELA/Reading and Math (3-8, HS); Science (5,8) 
Program 

Texas Texas Assessment of Knowledge and Skills, English and Math (3-11); Science 
Modified (5,8,10-11);Writing (4,7,10); Social Studies 


(8,10-11) (field tests included) 


i 


Note. Source: Albus et al. (2009). ELA = English Language Arts; HS = high school; 
4Still under development. Maryland plans to implement its alternate assessment based on modified academic achieve- 
ment standards in 2008-09 at the earliest. 


Participation Guidelines 


During both 2007 and 2008 the participation guidelines for all states included in this study 
indicated that a student must have an IEP to qualify for an AA-MAS (see Table 2). In 
other words, all states required that the student must already be identified as a student 
with a disability and be receiving special education services to participate in this assessment 
option. 

In 2007 only two states’ guidelines (Maryland, Oklahoma) indicated that a student’s previous 
performance on multiple measures must be considered, but 1 year later seven states (California, 
Connecticut, Maryland, North Carolina, North Dakota, Oklahoma, Texas) included this criterion. 
Examples of the types of multiple measures that states allowed included statewide assessments, 
district assessments, formal and informal classroom assessments, progress monitoring, and for- 
mative assessments. 

As shown in Table 2, in 2007 two of the six states with an AA-MAS had participation guidelines 
that specifically indicated that the student must be learning grade-level content, whereas | year 
later, the guidelines of seven of the nine states with an AA-MAS required that the student be 
learning grade-level content. In both 2007 and 2008, two thirds of the states with an assessment 
they considered to be an AA-MAS (e.g., four of the six states in 2007 and six of the nine states in 
2008) indicated that participation decisions should not be based on a student’s disability category 
label. 
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TABLE 2 
Alternate Assessment Based on Modified Academic Achievement Standards Participation Criteria 


State No. of 
Criterion Year CA? CT* KS LA MD NC ND OK TxX* States” 
Re ae ee re Fe ee 
Has IEP 2007 i Nes Kg cone eee 6 
DOORLERE. og Ne Nt ON ge he ai me a 9 
Previous performance on multiple measures 2007 x x 2 
2008 xX x Ka eX Ger oes 7 
Learning grade level content 2007 be, 2 
2008 XxX Xx eX i 
Not based on disability category label 2007 xX XX x x 4 
2008s Ne Nee xe es 6 
Not progressing at rate expected to reach grade level 2007 xX 1 
proficiency within school year covered by IEP 2008 xX Xx > ee OP. x 6 
Cannot demonstrate knowledge on regular 2007 Xx » P< x 4 
assessment even with provision of 
accommodations 2008 xX Xe Xx x Xx 6 
Receives specialized/ individualized instruction 2007 x Xa x 4 
2008 xX Xoo Xx x 5 
Not due to excessive absences, social, cultural, 2007 Xx Xx x 3 
language, economic, or environmental factors 2008 xX > ae, x 4 
Not receiving instruction based on extended or 2007 Xx 1 
alternate standards or not eligible to take 
AA-AAS 2008 XX Xe X a 
Previous performance on state assessment 2007 XS 2 
2008 xX ona x 4 
Receives accommodations during classroom 2007 Xx 1 
instruction 2008 Xx Xx Xe 4 
IEP includes goals based on grade-level content 2007 0 
standards 2008 xX x KEE XK 5 
Not based on placement setting 2007 Xx x 2 
2008 x Karly XS 3 
Not due to being identified as having a significant 2007 Xie ees 3 
cognitive disability 2008 Xx XC 3 
Performance multiple years behind grade level 2007 XBOX a 4 
expectations 2008 Xan 2 





Note. IEP = Individualized Education Program; AA-AAS = alternate assessments based on alternate achievement 
standards. 

“State did not have an alternate assessment based on modified academic achievement standards in 2007. ?2007, N = 
6; 2008, N = 9. 


In 2007 only one state (Maryland) of the six states with an AA-MAS indicated that for a 
student to participate in this assessment option, the student must not be progressing at a rate 
expected to reach grade-level proficiency within the school year covered by the IEP—however, 
1 year later the guidelines of six of the nine states offering this option included this criterion 
(California, Connecticut, Maryland, North Carolina, North Dakota, Texas). Likewise, there were 
large increases between 2007 and 2008 in the number of states that included a criterion which 
indicated that a student’s IEP must include goals based on grade-level content standards. No states’ 
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guidelines contained this criterion in 2007, but fives states (California, Connecticut, Maryland, 
North Carolina, North Dakota) included it in 2008. 

Two participation criteria were mentioned much less frequently in 2008 than in 2007. In 
2007, four of the six states that had an AA-MAS (Kansas, Louisiana, Maryland, North Carolina) 
included the “performance multiple years behind grade-level expectations” criterion. In 2008 
only two of the nine states (Kansas, Louisiana) included this criterion. Three of the six states 
with an AA-MAS in 2007 (Maryland, North Carolina, North Dakota) indicated that the decision 
was not due to the student being identified as having a significant cognitive disability, whereas in 
2008 the participation guidelines of only three of the nine states with an AA-MAS (Connecticut, 
Maryland, North Carolina) included this criterion. 


Participation Data 


We found participation data for the AA-MAS for five states (California, Kansas, Louisiana, North 
Carolina, and Oklahoma), but only one state had data across years. Thus, we decided to present 
data for the 2006-07 testing year, which is the year for which most states had data. The available 
data were found to vary widely in terms of the level of detail. 

Table 3 shows AA-MAS participation rates calculated for those states with data available. The 
numbers presented do not show the many ways in which states presented participation information. 


TABLE 3 
Average Participation Rates in States’ Alternate Assessment Based on Modified Academic Achievement 
Standards in Grades 4 and 8, 2006-07 


SS ou 


Reading/English Language Arts Mathematics 
State Grade 4 Grade 8 Grade 4 Grade 8 
gestion), Chen Gb i ie ee 
California’ 12,703 2.7% 10,702 2.3% 
Connecticut? 
Kansas° 816 2.5% 151.15 3.2% 816 2.5% 1,145 3.2% 
Louisiana? 653 1.3% 921 1.9% 653 1.3% 921 1.9% 
North Carolina® 3,263 3.1% 2,436 2.3% 2,838 Zale 2,531 2.3% 
North Dakota? 
Oklahoma 2,062 4.8% 2,365 5.6% 1,950 4.6% 2,358 5.6% 
Texas? 


a 

“California provided both “Students Tested” and “Students with Scores.” The former number was used as the 
numerator to calculate the participation rate. This rate might be higher than it would be had denominator been the 
total student population. Data were not available for Grade 8. Data were not available. “Kansas provided the number 
of students participating in the alternate assessment based on modified academic achievement standards (AA-MAS) 
at each grade, Grades 3-8 plus high school. 4] ouisiana provided the number of students in the AA-MAS and a 
participation rate based on the population of students with disabilities. We used the total student population to calculate 
rates. “North Carolina provided the number of students at or above Achievement Level III and the number of valid 
scores. We used the latter for the numerator. Student enrollment data were used as the denominator to calculate the 
participation rate. f Oklahoma provided students tested on the Oklahoma Modified Alternate Assessment Program 
with and without accommodations, as well as the number of all students. We used the latter numbers to calculate the 
participation rates. 
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California provided basic information on the number and percentage of students in Grades 3, 
4, and 5 who were assessed with the California Modified Assessment between 2006-07 and 
2007-08. Kansas provided data on the number and percentage of students assessed on the Kansas 
Assessment of Multiple Measures, which is its AA-MAS. It did this both for all grades combined 
and by individual grades, as well as disaggregated by content area (math and reading). Louisiana 
provided data on proficiency rates by content area (English Language Arts and Mathematics) 
for each grade, producing a total number assessed in the LAA2, its AA-MAS. North Carolina 
provided performance data for the EXTEND2, for each grade (3-8, plus 10), disaggregated by 
gender-ethnicity, economic disadvantage, limited English proficiency, migrant status, disability 
status, disability category, and parents’ education status. These data were provided at each grade 
for both mathematics and reading. Oklahoma provided data on its Oklahoma Modified Alternate 
Assessment disaggregated by whether the students used accommodations during the assessment. 
These data were provided along with the data for other assessments, for both reading and math, 
for Grades 3 to 8. 

Most states did not provide the total student population number in the same location as the 
AA-MAS data were presented. To provide comparable data across states, we located the total 
student population numbers and used them as denominators to calculate AA-MAS participation 
rates. As is evident in Table 3, which shows only Grades 4 and 8, states varied considerably in the 
calculated participation rates. California, Kansas, North Carolina, and Oklahoma had participation 
percentages that exceeded 2% of the total population. Participation rates ranged from 1.3% for 
Grade 4 reading/English language arts in Louisiana to 4.8% for reading/English language arts 
in Oklahoma. North Carolina, the only state with 2 years of data, had participation numbers in 
2006-07 that were higher than they had been in 2005-06. Specifically, the participation rates 
during the two school years (2006-07 vs. 2005-06) were Grade 4 Reading, 3.1% versus 2.2%; 
Grade 4 Math, 2.7% versus 2.6%; Grade 8 Reading, 2.3% versus 2.1%; Grade 8 Math, 2.3% 
versus 2.0%. 


Assessment and Question Characteristics 


The AA-MAS in most states included multiple-choice questions in both 2007 and 2008 (five of 
the six states offering this option in 2007 and all nine states with this option in 2008; see Table 
4). The AA-MAS in one state (e.g., Louisiana) included constructed responses in 2007; in 2008 
two states’ (Connecticut, Louisiana) assessments had constructed responses. The AA-MAS in 
four states (Kansas, Louisiana, North Carolina, Oklahoma) included a writing prompt in 2007. 
In addition to these four states, Texas (which did not offer an AA-MAS in 2007) had a writing 
prompt in 2008. One state’s assessment (North Dakota) required performance task responses. 
The format of North Dakota’s assessment was performance based in both 2007 and 2008—and a 
portfolio was required in 2007, but not required the following year. 


Test Design Changes 


The tests that states considered to be an AA-MAS differed in several ways from those states’ 
general assessments. Some states incorporated features that might be considered accommodations 
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TABLE 4 
Selected Assessment and Question Characteristics, 2007 and 2008 


i 





State 

Sa No. of 

Characteristic Year CA* Cr’ Koei’ Mo  NG:6| OUND OK TX States® 
Multiple choice 2007 Xx x x xX x 5 
2008 xX x x Xx xX xX xX xX x ®) 
Constructed response 2007 xX 1 
2008 x x 2 
Portfolio 2007 xX 1 
2008 0 
Performance task 2007 xX 1 
2008 x 1 
Writing prompt 2007 x Xx Xx x + 
2008 xX X xX Xx X 5 


SEU EEE EI EEEE EEE 


4State did not have an alternate assessment based on modified academic achievement standards (AA-MAS) in 2007. 
>The Maryland AA-MAS was under development in both years; however, publicly available information about the 
characteristics of the assessment was available. ©2007, N = 6; 2008, N = 9. 


on the general assessment into the design of their AA-MAS. As shown in Table 5, in 2007 two 
states (Kansas, North Carolina) with an AA-MAS incorporated the calculator into the AA-MAS 
assessment design; only one state (North Dakota) incorporated the calculator into the assessment 
design in 2008. 

In 2007 a distractor was removed in four of the six states offering the AA-MAS (Kansas, 
Maryland, North Carolina, and Oklahoma), in 2008 six of the nine states with this assessment 
option (California, Kansas, Maryland, North Carolina, Oklahoma, and Texas) removed a distrac- 
tor. For example, if a state’s general assessment had four answer choices for each multiple choice 
question, then on the AA-MAS there would be three choices. Four states (Kansas, Louisiana, 
North Carolina, Oklahoma) had fewer items on the AA-MAS than on the general assessment in 
2007; by 2008 eight states had fewer items (California, Connecticut, Kansas, Louisiana, Mary- 
land, North Carolina, Oklahoma, Texas). For North Dakota, the number of items on the regular 
assessment was not compared with the number of items on the AA-MAS due to the performance 
task format of that state’s AA-MAS. 

In 2007, three states (Kansas, Louisiana, North Carolina) had fewer reading passages on their 
AA-MAS than on the regular assessment; by 2008, five states (California, Kansas, Louisiana, 
North Carolina, Texas) had fewer reading passages. Three of the states with an AA-MAS had 
shorter reading passages in 2007 (Kansas, Louisiana, North Carolina). In 2008, in addition to 
these states, the assessments of two states (California and Texas) that offered an AA-MAS for 
the first time in that year contained this test design change. In 2007, four of the six states with 
an AA-MAS incorporated simplified language into the test design of the AA-MAS (Kansas, 
Maryland, North Carolina, and Oklahoma), and in 2008 six of the nine states with an AA-MAS 
incorporated simplified language (Connecticut, Kansas, Louisiana, North Carolina, Oklahoma, 
and Texas). 

In 2007, no states indicated that test questions were to be read aloud to all students who 
participated in the assessment; however, by 2008 one state (Texas) had incorporated this change 
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TABLE 5 
Selected Design Changes in States’ Alternate Assessment Based on Modified Academic Achievement 
Standards, 2007 and 2008 


Lio )e et br ge oe a ee 


_ State 


a No. of 


Selected Design Changes Year... CA@.. Cl? KS. LA. aMD? . NC). ND. OKs a0 Xt wstaicss 





Breaks as needed 2007 xX 1 
2008 x 1 
Calculator 2007 4 Xx 2 
2008 x 1 
Distractor removed 2007 x Xx x Xx 4 
2008 Xx x xX x Xx x 6 
Fewer items per page 2007 x x x 3 
2008 x Xx x x xX 5 
Fewer items on test 2007 x Xx x x 4 
2008 x x x x x x x x 8 
Fewer passages 2007 Xx x x 3 
2008 xX x Xx x x 5 
Key text bolded/underlined — 2007 x x Z 
2008 xX Xx Xx 3 
Larger font size 2007 x xX x 3 
2008 x xX ».4 x Xx 5 
Segmentation of passages 2007 x 1 
2008 Xx x x 8 
Shorter passages 2007 x 4 x 3 
2008 x x xX x x 5 
Simplified language 2007 Xx x x x 4 
2008 x x x x x x 6 
Test questions read aloud 2007 0 
2008 x 1 





4State did not have an alternate assessment based on modified academic achievement standards (AA-MAS) in 2007. 
>The Maryland AA-MAS was under development in both years; however, publicly available information about the 
characteristics of the assessment was available. “2007, N = 6; 2008, N = 9. 


into its AA-MAS design. Texas differentiated between oral administration of the test (which was 
allowed as an accommodation for the math and science assessments but not allowed on the reading 
test) and reading of the questions and answers (which was required as a routine administration 
procedure). For example, the test administrators’ manual for third grade for the reading test said, 
“Questions and answer choices will be read aloud to students by the test administrator’ (Texas 
Education Agency, 2008, p. 47). Test administrators were also directed to preread aloud certain 
sections of the test that were identified by a “speaker icon.” 


SUMMARY AND CONCLUSIONS 


Three more states had an assessment they considered to be an AA-MAS in 2008 than in the 
previous year. In 2008 all nine states that had developed an assessment they believed to be an 
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AA-MAS had tests that included multiple-choice questions. For states that had an AA-MAS in 
both 2007 and 2008, there were many changes in the states’ AA-MAS participation guidelines 
across the 2 years. The criteria that we reviewed in 2007 had been developed prior to the release 
of the final regulations, and many of the changes appeared to have been made as an attempt to 
better align the criteria with the regulations. For example, according to the regulations: 


The student’s progress to date in response to appropriate instruction, including special education and 
related services designed to address the student’s individual needs, is such that, even if significant 
growth occurs, the IEP team is reasonably certain that the student will not achieve grade-level 
proficiency within the year covered by the student’s IEP. (Final Rule 34 CFR Parts 200 and 300, 
Section 200.1(e) (2) (1) (A), 2007) 


In 2007, the participation guidelines of only one of the six states (Maryland) that had an 
assessment they believed to be an AA-MAS included the criterion that the student must not be 
progressing at the rate expected to reach grade-level proficiency within the school year covered 
by their IEP. However by 2008, six of the nine states with an AA-MAS (California, Connecticut, 
Maryland, North Carolina, North Dakota, Texas) included this criterion. 

Other examples of changes in the eligibility criteria that appeared to be a response to the 
regulations was the sharp increase in the number of states that required the determination of 
student progress be based on multiple measures, that the IEP include goals based on grade-level 
content standards, and that the student receive accommodations during classroom instruction. 
Also, probably in response to the regulations, there were several criteria that states were much 
less likely to include in 2008 than in 2007. In 2007, four of the six states (Kansas, Louisiana, 
Maryland, North Carolina) with an assessment they considered to be an AA-MAS indicated that 
for a student to participate in an AA-MAS, his or her performance should be multiple years 
behind grade-level expectations; by 2008 this had decreased to only two of the nine states with 
an AA-MAS (Kansas, Louisiana). Similarly, four of the six states with an AA-MAS in 2007 
(Maryland, North Carolina, North Dakota) had a criterion that participation in an AA-MAS 
should not be due to a student being identified as having a significant cognitive disability; in 2008 
only three of the nine states with an AA-MAS (Connecticut, Maryland, North Carolina) had this 
criterion. 

Changes in participation rates between 2007 and 2008 were less clear, primarily because 
these data were not available. In fact, only five states publicly reported any data about their AA- 
MAS. Just one of these states had data across 2 years (North Carolina). It was only by seeking 
other data that we were able to produce comparable rates across States. The reporting on the 
AA-MAS appeared to be missing the intent of the April 2007 regulation, which required that 
states and districts report on the participation and performance of students with disabilities on 
the alternate assessment based on modified academic achievement standards, just as they do for 
other assessments used for accountability purposes. Both IDEA and NCLB require this public 
reporting (National Center on Educational Outcomes, 2007). 

There were also a number of test design changes between 2007 and 2008—though in many 
instances the percentage of states making a selected change varied little across years. States may 
be finding that some potential test design changes do not meet the needs of the group of students 
that qualify to participate. 
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DISCUSSION 


Six of the nine states (California, Connecticut, Kansas, Louisiana, North Carolina, Texas) that 
have been early implementers of the AA-MAS were states that had offered an out-of-level (OOL) 
testing option (VanGetson, Minnema, & Thurlow, 2004) until federal policies required assessing 
grade-level content standards. In some of these states that had offered OOL tests, policymakers 
and department of education staffers may have thought that there was a need to offer a new less 
difficult assessment as the OOL option was phased out. Extra care needs to be taken by states that 
may perceive the AA-MAS as a replacement for an out-of-level test to ensure that the AA-MAS 
maintains high-expectations for students who participate in this option. Most states that had never 
provided an out-of-level option have not developed an AA-MAS. 

States are not required to offer the AA-MAS assessment option, and there may be a number of 
factors related to instruction and assessment of the group of students who qualify for this option 
that could potentially affect the decision. This investigation clearly indicated that the landscape 
surrounding the AA-MAS is rapidly changing. Even in states that do not offer an AA-MAS, 
most students participate in an assessment used for accountability purposes. Rather than needing 
another assessment option, some states may instead have instructional issues (Filbin, 2008; 
Fincher, Ban, & Rogers, 2007). The 2007 regulations stated that “students who are assessed based 
on modified academic achievement standards [must] have access to the curriculum, including 
instruction, for the grade in which the students are enrolled” (Final Rule 34 CFR Parts 200 
and 300, Section 200.1(f) (2) (iii), 2007). However, some students who may qualify for this 
assessment option may not currently have access to grade-level content. For example, the Georgia 
Department of Education conducted a curriculum implementation survey to examine whether 
students who were persistently low performing had access to grade-level, standards-based content. 
Preliminary results from the study suggested that some students did not have the opportunity to 
learn grade-level content; the author concluded that “any assessment solution must be coupled 
with instructional support” (Fincher, 2008). 

States need to ensure that students who participate in an AA-MAS have access to grade- 
level content, and they must carefully consider how an AA-MAS may fit into their assessment 
system—including any intended and unintended consequences. Filbin (2008), in an analysis of 
the initial March 2008 federal peer review of states’ AA-MAS, noted that 


a coherent assessment system is necessary to ensure that every student receives instruction linked 
to the grade-level standards based on their unique learning characteristics, that student achieve- 
ment is adequately measured, and that all students are included in accountability systems. It 
is crucial for States to consider how learning differs across students and how the various as- 
sessments within the current system tap both content knowledge and process skills with respect 
to the grade-level standards to effectively assign students to the most appropriate assessments. 


(p. 6) 


States developing an AA-MAS are doing so within the context of their existing assessment 
systems. The “gray areas of assessment” (Almond et al., 2000) may still exist, and there needs 
to be more recognition of issues with the assessment system itself. The landscape surrounding 
the AA-MAS will probably continue to change for several years as more is learned about the 
advantages and limitations of this assessment option. Future analyses are needed to provide more 


CHANGING LANDSCAPE OF ALTERNATE ASSESSMENTS 509 


detailed investigations into the characteristics of AA-MAS. This study provides a descriptive 
baseline for such future studies. 
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Federal regulations concerning the development and implementation of alternate assessments based 
on modified achievement standards include a set of safeguards intended to ensure that eligible students 
have access to grade-level general curriculum. These regulations concerning curricular access and 
opportunity to learn for students with disabilities may prove difficult for educational researchers and 
policymakers to operationalize and evaluate. This article provides a historical and policy context for 
efforts to ensure curricular access. In addition, this article reviews research on potential indicators 
of (or ways of measuring) access to the general curriculum and opportunity to learn. Finally, best 
practices for facilitating and evaluating access to the general curriculum are discussed. 


To create additional flexibility in measuring the achievement of certain students with disabil- 
ities, the U.S. Department of Education introduced changes to regulations under the No Child 
Left Behind Act and the Individuals with Disabilities Education Act (IDEA) in April 2007. These 
changes permitted states to (a) create modified achievement standards and (b) develop aligned 
achievement tests that are challenging “for a small group of students whose disability has pre- 
vented them from achieving grade-level proficiency and who likely will not reach grade-level 
achievement in the same timeframe as other students” (U.S. Department of Education, 2007). 
These aligned achievement tests, or alternate assessments based on modified academic achieve- 
ment standards (AA-MAS), must measure a student’s mastery of grade-level content but may be 
less difficult than the general large-scale assessments administered by the state. In response to 
the AA-MAS guidelines, states will be expected to account for the ways in which they provide 
participating students with access to the general curriculum. In this article, we look at the ways 
access to the general curriculum has been defined and review the concepts’ historical, legisla- 
tive, and legal foundations. Building on this information, we consider possible components of 
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students’ educational experiences that might be examined to determine whether students who are 
eligible for an AA-MAS have had access and opportunity to learn the skills and concepts outlined 
in grade-level content standards. 

Providing students with disabilities with access to the general curriculum is one of the central 
objectives of the new AA-MAS policy. In fact, one potentially high-impact feature of the new 
AA-MAS guidelines are a set of safeguards intended to ensure that eligible students have access 
to grade-level content standards. For example, the regulations concerning AA-MAS indicate 
“the [Individualized Education Program (IEP)] of a student assessed on the modified academic 
achievement standards include(s) IEP goals that are based on the academic content standards for 
the grade in which the student is enrolled, and (is) designed to monitor the student’s progress 
in achieving the student’s standards-based goals” (§200.1(f)(2)Gi)(A,B)). Similarly, the regula- 
tions include language that suggests educators must ensure that students who take an AA-MAS 
have “access to the curriculum, including instruction, for the grade in which (they are) en- 
rolled” (§200.1(f)(2)(iii)). Furthermore, these regulations concerning the necessity of providing 
standards-based IEPs and access to the general curriculum for students with disabilities also ap- 
pear to re-introduce a concept—opportunity to learn (OTL)—that, despite its attractiveness, has 
proven hard for educational researchers and policymakers to operationalize and evaluate. OTL is 
intended to describe what is “taking place in schools and classrooms to support students’ learning 
and progress, particularly relative to new expectations for student performance” (Herman, Klein, 
& Abedi, 2000, p. 16). In this article, we take a closer look at OTL as a concept and discuss legal 
cases and legislation that have articulated OTL within the context of school reform and special 
education. 


LEGISLATIVE AND LEGAL FOUNDATIONS FOR CURRICULAR ACCESS 
AND OTL 


Achieving universal opportunity to learn and curricular access have been important considerations 
and long-standing goals in educational reform. A number of judicial rulings as well as federal 
legislation have created a foundation for these efforts (see Table 1). For example, in 1954, the 
Brown v. Board of Education decision publicly acknowledged that the existence of segregated 
schools and education facilities create conditions that were inherently unequal, implying that 
African American students often did not receive the same access and opportunities as their White 
peers. 

One of the first cases to explicitly consider the concepts of curricular access and OTL was 
Debra P. v. Turlington (1981). This case concerned achievement tests administered in Florida high 
schools, which students were expected to pass to obtain a diploma. In the Debra P. case, African 
American students who had failed these achievement tests challenged that the assessments were 
racially biased, given without adequate notice, and designed to re-segregate African American 
students into low-quality remedial programs (Buckendahl & Hunt, 2005). Moreover, the students 
and their families perceived the testing requirement as a violation of their rights under due process. 
Although the court found that Florida schools had provided adequate access to and instruction in 
the skills and content on the test, the ruling in Debra P. established that all students must have an 
equal opportunity to learn the material on high school graduation tests (Pullin & Haertel, 2008). 
Moreover, although the Debra P. case established a legal precedent for the need to align content 
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TABLE 1 
Significant Legislative and Legal Decisions Related to Curricular Access and Opportunity to Learn 





1954 Brown vy. Board of Education: Segregated schools and education facilities created unequal conditions that 
limited access and opportunities to learn for African American students. 

1964 Title IV of the Civil Rights Act outlawed federal funding for school systems that treated students 
differently based on race, ethnicity, or national heritage. 

1966 Equality of Educational Opportunity (“The Coleman Report”) found differences between predominantly 
White and African-American educational institutions. The report also found that low-SES 
African-Americans benefited from integrated classrooms. 


1981 Debra P. v. Turlington: All students must have an equal opportunity to learn material prior to being tested 
on high school graduation exams. 

1994 Goals 2000 Educate America Act defined opportunity to learn standards. 

2001 No Child Left Behind (NCLB) reauthorized some federal education programs with a focus on (a) 


standards and increased accountability for states, schools, and school districts and (b) disaggregated 
assessment results to ensure all students academic progress. 

2004 Individuals with Disabilities Education Act (IDEA) reaffirmed that all children with disabilities should 
receive free and appropriate public education in the least restrictive environment, and required public 
schools to create Individualized Education Programs that facilitated access the general curriculum for 
students with disabilities. 

2007 Alternate Assessments based on Modified Academic Achievement Standards (AA-MAS) were permitted 
after the U.S. Department of Education introduced changes to NCLB and IDEA. States and district 
must ensure that students who qualify for an AA-MAS have had access grade-level curriculum. 


i 


of curriculum and instruction with the skills and concepts covered on achievement tests, it also 
illuminated the difficulty of defining and efficiently examining student’s curricular access and 
educational opportunities (Buckendahl & Hunt, 2005; Pullin & Haertel, 2008). 

Federal policy also has influenced the concepts of OTL and curricular access. For example, 
Title IV of the Civil Rights Act of 1964 outlawed the administration of federal funds to school 
systems that were segregated or treated students differently based on race, ethnicity, or national 
heritage (Venezia & Maxwell-Jolly, 2007). In addition, 8402 of Title IV required the government 
to “conduct a survey and make a report to the President and the Congress, within two years of 
the enactment of this title, concerning the lack of availability of equal educational opportunities” 
for students in the nation’s public schools. Commissioned by the U.S. Office of Education in 
accordance with the Civil Rights Act, Equality of Educational Opportunity (Coleman, 1966; 
often referred to as “The Coleman Report”) was intended to examine the similarities and dif- 
ferences in the programming and resources in schools that served either predominantly White 
and African American populations. The report included data from 600,000 students and teachers 
across the nation and indicated that there was a very strong difference in the quality of edu- 
cational institutions mainly attended by African Americans. Furthermore, the study indicated 
African American students from low socioeconomic status (SES) households appeared to benefit 
academically from integrated classrooms (Kiviat, 2000; Venezia & Maxwell-Jolly, 2007). 

Since the landmark Civil Rights Act, lawmakers have implemented federal policies intended 
to reduce educational inequities. Nearing the end of his term, president George H.W. Bush signed 
the Goals 2000 Educate America Act, which highlighted a number of policy goals that the nation’s 
schools would accomplish by the year 2000. Goals 2000 specifically addressed opportunity to 
learn standards and defined them as “the criteria for, and the basis of assessing the sufficiency or 
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quality of the resources, practices, and conditions necessary at each level of the education system 
to provide all students with the opportunity to learn the material in voluntary national content 
standards or state content standards” (§3(a)(7)). Within this federal policy, the development of 
OTL standards was intended to facilitate equity in school resources and programming (Carleton, 
2001). The proposed OTL standards, however, resulted in considerable disagreement among 
stakeholders and legislators. Some interest groups claimed Goals 2000's provisions were too weak 
to address the massive inequities among the nation’s schools and school districts, whereaes others 
criticized the potential costs of providing the material resources necessary to fully implement 
the OTL standards. Because of these disagreements, the OTL standards eventually were made 
“voluntary” in the reauthorization of the Elementary and Secondary Education Act (Bolt & Roach, 
2008; Ysseldyke, Thurlow, & Shin, 1995). 


THE CHALLENGE OF CURRICULAR ACCESS AND OTL IN THE 
CONTEXT OF AA-MAS 


The current movement to develop and implement AA-MAS eligibility guidelines and assessment 
strategies places enormous pressure on states to identify methods for facilitating and evaluating 
curricular access and opportunity to learn for students with disabilities. As the nonregulatory guid- 
ance indicated, states should “ensure that students have access to grade level content before they 
are assessed . .. and receive instruction in grade-level content after they are assessed” (U.S. De- 
partment of Education, 2007). Verifying students’ curricular access presents a challenge for edu- 
cators, policymakers, and researchers interested in supporting the implementation of an AA-MAS. 

A variety of features and components of the educational system may be evaluated to ensure 
curricular access and OTL for students who qualify for an AA-MAS (Figure 1). The OTL standards 
previously outlined in Goals 2000 are one starting point for developing evaluative framework. 
In the sections that follow, we consider the viability of some of the previously proposed OTL 
standards including 


¢ Alignment of curriculum, instructional practices, and assessments with content standards 
e Teachers’ capability 
e Access to instructional materials and technologies 


In addition, we consider and expand upon the constructs of student engagement and academic 
engaged time. Multiple researchers have “proposed to use these as an index of (opportunity to 
learn)” (Ysseldyke et al., 1995, p. 5). Finally, we review the potential effectiveness of two tactics 
unique to special education: (a) including students with disabilities in a general education setting, 
and (b) developing standards-based IEP goals. 


ALIGNMENT OF CURRICULUM, INSTRUCTIONAL PRACTICES, AND 
ASSESSMENTS WITH CONTENT STANDARDS 


One approach to facilitating OTL and curricular access to the general curriculum is by examining 
the extent to which the content of instructional programs, state content standards, and assessments 
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WAYS OF 
CONCEPTUALIZING POSSIBLE MEASURES 


Designing/implementing 
alignment studies using the 
“Webb", Surveys of Enacted 

Curriculum (SEC), and the 
Achieve models. 


Alignment of curriculum, 
instructional practices, 
and assessments with 

content standards 


Identify and measure four subtypes of 
Student engagement and student engagement: academic , 
academic engaged time behavioral, cognitive, and affective 
(Reschly and Christenson, 2006). 


Collect information on teacher training, 
Teacher capabilities knowledge, and skills to facilitate 


Conceptualizing and curricular access and OTL. 


Measuring Opportunity to 
Learn & Curricular Access 


Collect information on the 
classroom/school environment (e.g., 
Access to instructional collaboration between special and general 
materials and technologies educators), availability of curricular 
resources, and use of assistive and 
instructional technology. 


Data on time in general education 
Inclusion in the general classrooms and provision of 
education classroom instructional accommodations 
and/or modifications. 


In formation on the quantity and 
Standards-based IEPs qaulity of IEP goals based on the 
grade-level content standards. 





FIGURE 1 Ways of conceptualizing and measuring opportunity to learn (OTL) and curricular access. Note. 
IEP = Individualized Education Program. 


correspond to one another. Standards-based reform and accountability programs generally assume 
that educational improvement demands coordination in the development and implementation 
of three components of the educational environment: curriculum, instruction, and assessment 
(Elliott, Braden, & White, 2001; Webb, 1997, 2002). The degree to which these components 
work together to facilitate educators’ work and students’ learning is often referred to as alignment 
(Roach, Niebling, & Kurz, 2008). 

Alignment is an area of policy and practice that has the potential to have a positive impact on 
all students’ learning and achievement, including students who are eligible to take an AA-MAS. 
Certainly, if students with disabilities are not provided instruction on the skills and concepts on 
general grade-level assessments, they cannot be reasonably expected to demonstrate “proficiency” 
on a standards-based assessment (Roach et al., 2008). 

A number of frameworks have been developed to evaluate the alignment among content stan- 
dards, classroom instruction, and achievement tests. The Council of Chief State School Officers 
(2002) identified three “preferred” models for use in designing and implementing alignment 
studies: (a) the “Webb” model, (b) the Surveys of Enacted Curriculum (SEC) model, and (c) the 
Achieve model. Each model consists of a series of indices that provide a summary of the general 
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match or overlap between skills and knowledge presented in standards, assessments, and (in some 
cases) classroom instruction (Resnick, Rothman, Slattery, & Vranek, 2003; Roach et al., 2008). 
Because of its inclusion of data on classroom instructional practices, the SEC model holds the 
most promise for determining the extent to which students who participated in an AA-MAS have 
had access to the general grade-level curriculum. 

In the SEC model, teacher surveys are used in the collection information regarding the content 
of classroom instruction (i.e., the “enacted curriculum,” the content of what teachers teach). The 
SEC teacher surveys ask teachers to describe the skills and concepts taught in their classrooms, 
and typically are completed at the end of the school year. Classroom teachers rate the amount of 
instructional emphasis (e.g., how many lessons taught) and levels of cognitive demand expected 
(i.e., complexity of tasks) for various curricular topics (Porter, 2002). Results from teacher 
surveys provide data that can be converted to proportions that describe the amount of time 
spent on each topic and at each level of cognitive demand. The data can then be compared to 
the content covered by assessments, standards, or instructional support materials and used to 
calculate alignment indices for evaluating curricular access or OTL (Roach et al., 2008). 

Using the SEC (or a similar evaluative method) might allow states and school districts to gather 
descriptive data on classroom instruction and to examine the alignment of classroom instruction 
with skills and concepts outlined in grade-level standards and measured by large-scale assessment 
instruments. The alignment index (based on the SEC’s method) could also be used to quantify 
the content of classroom instruction provided to students with disabilities, thereby facilitating a 
direct quantitative analysis of how instructional content impacts their academic progress. In one 
study, Gamoran, Porter, Smithson, and White (1997) found that increased alignment between the 
classroom instruction and skills measured by a standardized mathematics test were associated 
with gains in achievement for low-achieving, low-socioeconomic high school students. In the 
same study, however, Gamoran and colleagues found that students in remedial classes received 
less standards-aligned instruction and typically had smaller gains in achievement scores when 
compared to their peers in general grade-level courses (Roach et al., 2008). 

Two groups of researchers (Karvonen, Wakeman, Flowers, & Browder, 2007; Roach & Elliott, 
2006) have attempted to modify or extend the SEC model for use with students with significant 
cognitive disabilities who take an alternate assessment based on alternate academic achievement 
standards (AA-AAS). Roach and Elliott created a curricular access questionnaire that included 
items specific to one states’ alternate assessment and extended curriculum framework for stu- 
dents with disabilities. This curricular access questionnaire asked whether particular performance 
indicators (i.e., objectives and rating scale items) were part of a student’s curriculum during the 
current or any previous school year. The curriculum access questionnaire was completed by 
students’ teachers and was subsequently included as one predictor of student performance in a 
study to examine the influence of curricular access on alternate assessment scores. Karvonen 
and colleagues (2007) developed a teacher self-report measure, Curriculum Indicators Survey 
(CIS), for use in determining the extent of curricular access for students with significant cognitive 
disabilities from preschool through 12th grade. The CIS includes two sections: (a) questions 
regarding teachers’ experiences, training, classroom characteristics, instructional materials, and 
assessment strategies, which the teacher answers in regard to their students, and (b) ratings of the 
intensity of instructional coverage for various topics in English/Language Arts and Mathematics, 
which are completed regarding the educational program provided to a particular student. The 
development and initial implementation of these instruments point to the potential utility of the 


FACILITATING AND EVALUATING GENERAL CURRICULUM ACCESS O17 


SEC model for evaluating access and exposure to the general curriculum for students who are 
eligible to take an AA-MAS. 


STUDENT ENGAGEMENT AND ACADEMIC ENGAGED TIME 


Alignment methodologies capable of assessing the match between classroom instruction and 
content standards can provide a useful avenue for determining students’ access to the general 
education curriculum as well as the OTL the content and skills on large-scale assessments (e.g., 
states’ AA-MAS). One aspect missing from this approach, however, is the degree to which students 
are actually engaged with the enacted curriculum. Mere exposure to a well-aligned curriculum 
does not guarantee student engagement with instructional materials or in classroom activities, 
and without engagement students with disabilities cannot realize whatever opportunities may be 
provided to them. As such, educators’ efforts to facilitate and evaluate OTL and curricular access 
for students who take an AA-MAS also should account for current developments in research on 
student engagement. 

As noted by Furlong and Christenson (2008), consensus has been reached on conceptualizing 
student engagement as a multidimensional construct that accounts for students’ thoughts, feelings, 
and behaviors (Fredericks, Blumenfeld, & Paris, 2004; Furlong et al., 2003). Affective, behavioral, 
and cognitive dimensions of engagement (e.g., Finn, 1989) exist alongside academic dimensions 
such as time on task or academic learning time (e.g., Gettinger, 1986). The latter has enjoyed 
sustained interest among practitioners for its consistent association with student achievement 
(Denham & Lieberman, 1980; C. W. Fisher & Berliner, 1985; Gettinger & Seibert, 2002). 
Academic learning time is defined as the amount of instructional time during which students 
are actively and productively engaged in learning (C. W. Fisher & Berliner, 1985). However, the 
academic dimension of student engagement was only recently incorporated into the construct’s 
common three-part typology (e.g., Reschly & Christenson, 2006). Because all four dimensions 
can be considered relevant to provide students who participate in an AA-MAS with substantiate 
instructional opportunities, we briefly discuss each dimension. 

Despite definitional and conceptual differences, research on student engagement has largely 
moved away from considering engagement a dichotomous student attribute and instead regards 
the dimensions of engagement to be mediated by contextual factors such as family, peers, and 
school (Appleton, Christenson, Kim, & Reschly, 2006). Reschly and Christenson’s (2006) inte- 
grative model defines four subtypes of student engagement. Indicators of academic engagement 
include time on task, credits earned toward graduation, and amount of homework completed. 
Behavioral engagement pertains to indicators such as attendance, classroom participation, ex- 
tracurricular activities, and utilizing extra credit options. Indicators of cognitive engagement refer 
to self-regulation, goal setting, utility value of school to future aspirations, and strategizing. Af- 
fective engagement comprises indicators such as school membership, identification with school, 
and sense of belonging. All four dimensions of student engagement can impact academic out- 
comes (e.g., AA-MAS results) as well as social and emotional competence (e.g., interpersonal 
relationships, conflict resolution skills). 

The four-part typology brings together research linking indictors of cognitive and affective 
engagement to positive learning outcomes (National Research Council & Institute of Medicine, 
2004), and research linking academic learning time to student achievement (C. W. Fisher & 
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Berliner, 1985). Overall, research has found student engagement to be a strong predictor of 
student achievement and behavior in school (e.g., Mounts & Steinberg, 1995; Voelkl, 1995), 
including higher grades and test scores (€.g., Willingham, Pollack, & Lewis, 2002; Roderick & 
Engle, 2001). Student engagement is further linked to lower dropout rates, and many researchers 
consider it the most promising theoretical model for understanding and addressing school dropout 
rates (e.g., Appleton et al., 2006; Finn, 1989). Last, student engagement has applicability to all 
students, regardless of ethnicity, SES level, or disability status. For instance, recent data from the 
2006 High School Survey of Student Engagement indicated that approximately 28% of the more 
than 80,000 respondents across a wide range of geographic and demographic factors reported 
being unengaged in school (Yazzie-Mintz, 2007). Students were more likely to exhibit lower 
levels of engagement, if they were male, from an ethnic group other than White or Asian, from 
lower SES families and/or communities, or (most relevant for this discussion) received special 
education services. 

The extent to which student engagement is applicable to evaluating curricular access and 
OTL is largely dependent on the definition of these constructs and the inferences to be drawn 
from potential evaluative strategies. Whether all or some of the discussed dimensions of student 
engagement should be incorporated into a measure of curricular access and OTL is subject to 
debate and will, at least in part, depend on practical considerations such as the availability of 
personnel and financial resources. 


TEACHER CAPABILITIES 


Effective curricular access requires close examination of teacher training, as well as the relevant 
knowledge and skills teachers possess. Teachers, after all, play a critical role in ensuring OTL 
and curricular access for students who take the AA-MAS. Although standards-based reform has 
created significant movement toward unified rigorous student achievement outcomes, teacher 
certification requirements, preservice training programs, and access to quality professional de- 
velopment continue to vary across states and school districts (Darling-Hammond, 1999). Some 
states, however, have begun adopting rigorous teaching standards like those put forth by the 
Interstate New Teacher Assessment and Support Consortium and the National Council for Ac- 
creditation of Teacher Education (Wayne & Youngs, 2003). Attention to the quality of teacher 
education is particularly important because research has shown that adequate preparation and cer- 
tification can positively impact teacher performance and effectiveness (Darling-Hammond, 1999; 
Darling-Hammond, 2000), and facilitate improvement in specific areas of student achievement 
(e.g., mathematics; see Wayne & Youngs, 2003). 

Buell, Hallam, Gamel-McCormick, and Scheer (1999) indicated that “the meaningful edu- 
cation of students with and without disabilities relies on a partnership between [both special 
and general] educators” (p. 144). Indeed, minimal teacher collaboration and a need to clarify 
general and special educators’ roles in planning and implementing instruction may hinder cur- 
ticular access for students with disabilities, including those who take an AA-MAS (Buell et al., 
1999; Jackson, Harper, & Jackson, 2001; Otis-Wilborn, Winn, Griffin, & Kilgore, 2005). Parallel 
training programs for general and special educators often have resulted in separate approaches 
to education, without sufficient programming to ensure general educators’ buy-in to inclusive 
practices (Buell et al., 1999). General educators may express concern over insufficient support 
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in meeting the needs of students with disabilities in their classrooms, especially when it comes 
to appropriately adapting materials and handling behavioral issues (Agran, Alper, & Wehmeyer, 
2002; Buell et al., 1999). By the same token, special educators often cite resistance from their 
general education colleagues as a significant barrier in ensuring students’ OTL and curricular 
access (Agran et al., 2002). Several researchers have called for a more collaborative approach 
to educating students with disabilities by including both general and special educators in the 
planning and decision-making process, as well by providing more training and material supports 
for teachers (Agran et al., 2002; Buell et al., 1999; Jackson et al., 2001). Buell and colleagues 
concluded that providing appropriate resources and professional development for general edu- 
cation teachers will likely increase teachers’ perception that they positively impact the school 
achievement of students with disabilities (Buell et al., 1999). In addition, special educators do not 
always have adequate knowledge of the general curriculum to provide the consultative support 
that would facilitate curricular access for students who take an AA-MAS (Otis-Wilborn et al., 
2005). 

A related issue identified by teachers and researchers concerns paraeducators who are expected 
to provide instructional support to students with disabilities in the classroom, but may not be 
sufficiently trained to facilitate curricular access (Agran et al., 2002). Therefore, all educators 
who provide instruction to students with disabilities should have the opportunity to participate 
in professional development regarding OTL and access to the general curriculum. To meet 
the AA-MAS regulations, states may need to explore methods of monitoring (e.g., surveys of 
teachers and paraprofessionals, review of course syllabi and training materials) the extent to 
which curricular access for students with disabilities is addressed in teacher preparation and 
professional development programs. 


ACCESS TO INSTRUCTIONAL MATERIALS AND TECHNOLOGIES 


Effective curricular access is also dependent upon a school and classroom environment that 
supports it, including access to sufficient instructional resources. For example, the results of a 
recent study indicated special education teachers struggled to get access to the same curricular 
materials provided to their general education colleagues and that their physical location away 
from general education classrooms hindered efforts to provide access to the general education 
classroom and curricula. Students with disabilities had limited opportunity to interact with their 
peers, and the special educators found it difficult to interact and consult with each other and 
with general education teachers (Otis-Wilborn et al., 2005). Schools can help mitigate this by 
promoting shared, collaborative instructional contexts and by allowing time for shared planning 
between general and special educators (Jackson et al., 2001). To more fully understand curricular 
access for students participating in AA-MAS, states and school districts might gather data on the 
prevalence of collaboration and consultation between general and special educators. 

Providing access to instructional and assistive technology also may help students with disabili- 
ties access general education instruction and curriculum in multiple ways (e.g., increasing student 
participation, providing more communication options, and creating cross-curricular connections; 
D. Fisher & Frey, 2001). Smith and Jones (1999) found that educators’ discomfort with the use 
of assistive technology limited the extent to which students had access to it. They suggested 
that states and school districts provide educators with training in using such technologies as an 


520 A. T. ROACH ET AL. 


educational tool. Recent research by the National Alternate Assessment Center demonstrates the 
necessity of professional development in this area (Kleinert, Towles-Reeves, Kearns, & Kleinert, 
2007). In a survey of educators regarding their students’ (n = 7,075) expressive and receptive 
communication, Kleinert and colleagues found that less than 50% of students with disabilities 
at a pre- or emerging symbolic communication level had access to augmentative and alternative 
communication devices. Although these students would be unlikely to take an AA-MAS, the lack 
of basic communication and instructional supports for this group calls into question the availabil- 
ity and use of assistive technology (e.g., transcribing software, audiobooks, calculators) for other 
students with disabilities. Collecting information regarding the implementation of assistive and 
instructional technology in general and special education contexts may be an important aspect of 
ensuring curricular access and opportunity to learn for students who take an AA-MAS. 


INCLUSION IN THE GENERAL EDUCATION CLASSROOM 


One possible approach to facilitating OTL and curricular access for students participating in AA- 
MAS would be to monitor time spent in inclusive educational settings (€.g., general education 
classrooms). The IDEA (1997) required states and school districts to guarantee that “to the 
maximum extent appropriate, children with disabilities . . . are educated with children who are 
nondisabled” ($612, 11 Stat 61). The most recent reauthorization of IDEA expanded expectations 
for students with disabilities beyond mere presence in general education classrooms toward full 
participation with and mastery of academic content (Browder et al., 2007). As such, students who 
qualify for an AA-MAS need access to the instruction provided in general education classrooms 
with accommodations and/or modifications that will allow them to progress towards grade-level 
content and achievement standards. 

Researchers have identified some significant trends pertaining to the effectiveness of inclusive 
classrooms and segregated pull-out programs for students with disabilities. Wehmeyer, Lattin, 
Lapp-Rincker, and Agran (2003) conducted an observational study of middle school students with 
mental retardation to determine amount of time that these students spent working on tasks that 
were linked to academic standards in both the special education and general education settings. 
Data collected showed that students with disabilities who were included in general education 
classrooms worked on tasks related (or “linked’’) to a content standard during 90% of observation 
intervals. Conversely, students with disabilities in segregated settings were engaged in tasks 
linked to content standards only 50% of the time (Wehmeyer et al., 2003). These results imply 
that (for students with mental retardation) inclusion in the general education classroom appeared 
to facilitate more time spent working on instructional tasks that are linked directly to state or 
district grade-level standards. 

Based on their review of inclusion literature, D. Fisher, Roach, and Frey (2002) argued that 
“there is considerable evidence over the last 15 years to suggest that segregation of students in 
special education in separate classes is actually deleterious to their learning and that students in 
special education generally perform better on average in a regular classroom” (p.71). Forexample, 
Fisher and colleagues cited a 1995 study by Stevens and Slavin that examined the performance of 
76 students with learning disabilities. Forty of the students received their education in inclusive 
general education classrooms and 36 received their education in a pull-out service delivery model 
without inclusion. Although their were no significant pretest differences in reading/language 
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arts or mathematics performance (as measured by standardized tests), students in the inclusive 
classrooms demonstrated significantly higher posttest performance in reading vocabulary (F = 
13.48, p < .01), reading comprehension (F = 14.39, p < .01), language expression (F = 11.41, 
p < 01), math computation (F = 10.77, p < .01), and math applications (F = 3.75, p < .05). 

Waldron and McLeskey (1998) compared the performance of 71 elementary students with 
learning disabilities educated in the inclusive settings with 73 students who were educated in 
the segregated settings. Pretest results indicated the two groups did not differ significantly in 
reading and mathematics scores or cognitive ability. Posttest results, however, indicated that 
students in inclusive classrooms made significant gains in reading compared to students in pull- 
out programs (in some cases approaching the rate of growth for their peers without disabilities). 
The same was not true in mathematics, however, where no significant differences were found 
between the achievement of the two groups of students with disabilities. One reason for these 
conflicting findings may be a lack of specialized instruction provided in some inclusive settings. 
In other words, access to general education classrooms may not be enough to produce changes in 
student performance. In their case studies regarding the educational experiences of five students 
with learning disabilities, Baker and Zigmond (1995) found that students with disabilities did 
appear to make some academic progress, but they often did not consistently receive instruction in 
inclusive settings that was specialized to address their individual educational needs. The findings 
from these two studies suggest that collecting data on the need for and provision of individualized 
instruction and modifications may be important in understanding the effectiveness of inclusion 
in general education contexts for students who take an AA-MAS. 

Salend and Duhaney (1990) cited a survey conducted by the National Center for Restructuring 
and Inclusion (1995) that addressed the impact of inclusion on students with and without disabil- 
ities. This survey of school districts suggested that schools providing inclusive classrooms were 
more likely to report increased academic gains for students with disabilities, including improved 
standardized test scores, mastery of goals set forth in IEPs, improved grades, and increases in 
attentive and pro-social behavior. These results are promising because they indicate that inclu- 
sion in general education settings is perceived as beneficial by many educators. Unfortunately, 
these survey results can provide only the suggestion, rather than confirmation, that time spent in 
general education classrooms leads to improved access, opportunities, and outcomes for students 
with disabilities. In their review of the inclusion literature, D. Fisher et al. (2002) indicated that 
“surprising few studies have investigated the effect of inclusion on the academic achievement of 
students with disabilities” (p. 72). Thus, time in inclusive settings may be viewed as a precondi- 
tion for providing OTL and access to the content standards for students who take an AA-MAS, 
but it is probably insufficient as an indicator of these outcomes. 


STANDARDS-BASED IEPS 


Another option for ensuring curricular access and opportunity would be promoting the devel- 
opment of standards-based IEP goals for students who take an AA-MAS. The most recent 
reauthorization of IDEA requires that students’ IEPs have (a) descriptions of a students’ present 
level of educational performance, including how his or her disability affects involvement and 
progress in the general curriculum; (b) measureable annual goals referenced to grade-level gen- 
eral curriculum; (c) description of modifications or supports needed to facilitate access to and 
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progress in the general curriculum; and (d) a description of the extent that the student will partic- 
ipate in the general education classes and activities. Karger (2004) suggested that “the IEP can 
be viewed as the central mechanism, both legally and educationally, for ensuring access to the 
general education curriculum” (p. 6). The centrality of the IEP document and process, however, 
requires that IEP team members are prepared to plan and facilitate (a) students’ access to the 
general education classroom, (b) involvement with general education instructional materials and 
activities, and (c) progress toward understanding the skills and concepts in the general education 
content standards (Bolt & Roach, 2008; Hitchcock, 2001). 

Unfortunately, existing research suggests that the current IEP process does not result con- 
sistently in improved access and involvement in the general education curriculum. In their case 
studies of students with disabilities, D. Fisher and Frey (2001) discovered a “disconnect” be- 
tween students’ IEPs and the curriculum and instruction provided in general education class- 
rooms. Teachers and family members both reported the goals and objectives included in students’ 
IEPs did not match the curriculum and instruction provided to students in inclusive environ- 
ments. Fisher and Fry’s findings are consistent with other research on IEPs that found minimal 
coordination between special educators and general educators (Lipsky & Gartner, 1997) and dif- 
ficulties developing measurable goals and objectives for student progress in the general education 
curriculum (Yell, 1998). 

A recent study (Karvonen & Huynh, 2007) examining the relationship of the content of 
students’ (n = 292) IEP objectives and achievement, as measured on an AA-AAS, provides some 
insights into using standards-focused IEP goals as an index of curricular access and opportunity 
to learn. Karvonen and Huynh found students who scored proficient on the AA-AAS in reading 
were more likely to have academic-focused IEP objectives than those who did not achieve 
proficiency (47% vs. 30%). Similarly, more of students who scored proficient on the AA-AAS in 
mathematics also had academic objectives on their IEPs (48% vs. 33%). Students who had IEPs 
that addressed reading comprehension content standards were more likely to score proficient 
(94% proficient or above) in English/Language Arts than their peers without similar content 
coverage (73% proficient or above). This relationship was not observed in mathematics, students 
with coverage of numbers and operations standards in their IEP goals were no more likely to 
be proficient in mathematics on the AA-AAS. These findings must be interpreted with caution, 
however, because the majority of the students’ IEP goals were rated as being at the lowest level of 
cognitive complexity (i.e., DOK 1). Moreover, approximately 50% of students did not have IEP 
goals that addressed reading comprehension objectives, and more than 33% did not have a goal 
that addressed number concepts objectives. Thus, it may be that the provision of standards-based 
IEP goals is related to students’ pre-existing ability levels or the severity of their disabilities. 

A survey by the National Association of State Directors of Special Education (1999) of state- 
level administrators (n = 33) identified numerous difficulties in using students’ IEPs as tools for 
facilitating access to the general curriculum. According to survey respondents, 


1. IEP goals and objectives tended to address reading and mathematics, but not other subject 
domains (e.g., social studies, science). 

2 State content standards were viewed as too broad and complex to serve as a focus for students’ 
IEP goals and objectives. 

3. Parents and teachers needed additional information and training on how state standards apply 
to all students. 
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4. Both special and general educators needed professional development on how to link IEP goals 
and objectives to state content standards, large-scale assessments, and general curriculum 
materials. 

5. Both special and general educators needed access to and professional development on 
curriculum-based assessments for tracking students’ progress on IEP goals and objectives 
that are linked to the general curriculum (Bolt & Roach, 2008; Karger, 2004). 


For the IEP to serve as the primary means of facilitating access to the general curriculum 
and improve academic outcomes for students who take an AA-MAS, states and school districts 
need to provide extensive professional development and increased monitoring to ensure that the 
goals and support services outlined in IEP documents are implemented as planned. In short, the 
requirement for educators to address general education content standards in IEP goals of students 
who take an AA-MAS represents a promising but not yet validated strategy for facilitating OTL 
and access to the general curriculum. 


BEST PRACTICES IN MEASURING CURRICULAR ACCESS AND 
OPPORTUNITY TO LEARN FOR STUDENTS WHO ARE ELIGIBLE FOR 
AN AA-MAS 


Although the federal guidelines for AA-MAS programs clearly address the need to create 
standards-based IEPs, establishing each of the components outlined in the previous sections 
may be necessary to create “true” access and opportunity to learn for students with disabilities. 
In fact, an argument could be made that the students eligible to participate in an AA-MAS 
(i.e., “persistently low-achieving students”) require access to enriched instructional resources, 
well-organized and safe classrooms, and the most skillful teachers to experience success with 
and progress in the general grade-level curriculum. At present, it is unclear whether the federal 
government, state departments of education, and local school districts have the political will and 
economic resources necessary to truly create curricular access and opportunity to learn for this 
group of students. 

Because curricular access and opportunity to learn are such complex constructs, states and 
school districts are faced with a disorienting range of possibilities for facilitating and evaluating 
progress in these areas. Moreover, an additional concern for policymakers and educational leaders 
is the feasibility and technical adequacy of the various evaluative strategies. Although the previous 
sections provide some guidance for making decisions in this area, we would like to propose two 
approaches that we believe hold promise: 


1. Development and validation of a measure of IEP quality. 
2. Multidimensional measurement of students’ engagement in general education classrooms and 
instruction. 


In light of the centrality of standards-based IEP goals in the AA-MAS regulations, states 
and school districts should pursue efforts to evaluate the quality of students’ IEP goals and 
their influence on curricular access and subsequent AA-MAS performance. As part of a current 
General Supervision Enhancement Grant [the Consortium for Modified Alternate Assessment 
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Development and Implementation (CMAAD)D)], the Indiana State Department of Education and 
researchers from Vanderbilt University and Georgia State University are currently engaged in 
the development of a tool for evaluating standards-based IEP goals. Creation of the CMAADI 
IEP tool was guided by Project Forum’s Seven-Step Process to Creating Standards-based IEPs 
(Davis Holbrook, 2007) and an IEP development training module created by the Individualized 
Classroom Accountability Network Project. The CMAADI IEP tool provides a rubric to evaluate 
each IEP goal in the following areas: (a) addressing students’ identified educational needs, (b) 
aligning to grade-level content standards, (c) identifying present level of performance, and (d) 
including adequate progress monitoring strategies. The current validation plan includes examining 
the relationship between the quality of IEP goals and curricular access (as measured on the CIS; 
Karvonen & Huynh, 2007) and performance on large-scale assessments. In addition, cases will 
be drawn from schools that have received Individualized Classroom Accountability Network 
Project training on IEP goal development as well as from comparison schools that have not 
yet received training. This will allow us to examine the effectiveness of training and access to 
professional resources regarding standards-based IEPs in facilitating curricular access for students 
with disabilities, including students who qualify for an AA-MAS. 

If the latest AA-MAS regulations are to be viewed as articulating the expectation of common 
academic content standards for all students (Cortiella, 2007), then assessment of student engage- 
ment should clearly be incorporated into measures of curricular access and opportunity to learn. As 
previously noted, there is a long history of research (e.g., time-on-task, academic learning time) 
that suggests productive engagement with meaningful instructional tasks promotes academic 
achievement. Measurement of student engagement might differ depending on the dimensions of 
the construct being assessed. Cognitive and affective engagement feature less observable factors 
such as self-regulation, perceived utility value of school, and goal setting. Academic and be- 
havioral engagement yield more overt indicators including attendance, time on task, and credits 
earned toward graduation. The former is thus more amenable to self-report measures, whereas the 
latter can draw from teacher reports, record reviews, and direct classroom observations. Examples 
and combinations of each option are available (e.g., Academic Competence Evaluation Scales, 
DiPerna & Elliott, 2000; Children’s Engagement Questionnaire, McWilliam, 1991; Engagement 
Check II, McWilliam, 1999; HSSSE, Yazzie-Mintz, 2007; Student Engagement Instrument, Ap- 
pleton & Christenson, 2004). However, examination of engagement subtypes in isolation, use of 
different informants, amalgamation of constructs of engagement, and issues related to inferences 
based on observation have resulted in concerns about psychometric soundness and construct 
validity of engagement measures (Appleton et al., 2006; Jimerson, Campos, & Greif, 2003). We 
believe the current AA-MAS provide a context and (perhaps) the resources for additional work 
to develop technically adequate measures of student engagement. 


CONCLUSION 


The development and implementation of states’ AA-MAS will allow a select group of students 
with disabilities to demonstrate their proficiency for accountability purposes. In addition, the 
AA-MAS policy appears intended to (a) facilitate students’ inclusion and OTL and (b) motivate 
special educators to provide access to the grade-level content standards. The most recent Standards 
for Educational and Psychological Testing indicates that “tests (e.g., AA-MAS) are commonly 
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administered in the expectation that some benefit will be realized from the intended use of scores 
.... A fundamental purpose of validation is to determine whether these specific benefits are 
likely to be realized” (American Educational Research Association, American Psychological 
Association, & National Council on Measurement in Education, 1999, p. 16). As states’ AA- 
MAS systems are fully developed and implemented, we believe careful consideration of a variety 
of indicators (e.g., students’ IEPs, test scores, teachers’ levels of training, etc.) will be necessary 
to provide compelling evidence regarding progress toward this goal. 

In the past, many researchers and policymakers have struggled to articulate and evaluate 
OTL and curricular access. Federal policy and educational research has, at different times, high- 
lighted various areas related to these two concepts including (a) alignment between standards, 
assessments, and instruction; (b) student time on task and engagement; (c) teachers’ professional 
preparation and access to resources; (d) inclusion in the general education classroom; and (e) 
development and implementation of standards-based IEPs. With the recent focus on curricular 
access as a precondition for participation in an AA-MAS, it is important to consider how historical 
precedent and existing research can help guide policymakers and educators in developing appro- 
priate evaluative strategies and support materials to achieve this goal. In addition, researchers 
should be encouraged and supported in developing and validating new strategies to evaluate the 
provision of OTL and access to the general curriculum for students with disabilities. 
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Federal regulations allow up to 2% of the student population of a state to achieve proficiency for 
adequate yearly progress by taking an alternate assessment based on modified academic achievement 
standards (AA-MAS). Such tests are likely to be easier, but as long as a test is considered a valid 
measure of grade level content, it is allowable as an AA-MAS (U.S. Department of Education, 
2007b). In this article, we examine procedures for developing, modifying, and evaluating items 
and tests using an evolving modification paradigm, as well as a classic reliability and validity 
framework. Theoretical influences, such as principles of universal design, cognitive load theory, and 
item development research, are discussed. The Test Accessibility and Modification Inventory, a tool 
that provides systematic and comprehensive guidance to help educators modify grade-level tests, 
is introduced. Cognitive lab methods and experimental field tests are then described, along with 
examples and key findings from each, relevant to AA-MASs. The article concludes with a discussion 
of precautions, lessons learned, and questions generated about the methods used to improve both 
access and test score validity for the students who are eligible for this new alternate assessment. 


This article describes a systematic process of item modification, founded in theory, individual 
item analysis through statistics and a modification tool, and pilot testing in a cognitive lab. Recent 
changes in federal policy allow a small percentage of students with disabilities to take an alternate 
assessment based on modified academic achievement standards (AA-MAS) to facilitate greater 
access to grade-level content standards. Such tests are expected to yield more accurate measures of 
these students’ achievement. The intended results of increased access to the standards are twofold. 
The first expectation is that students are able to demonstrate what they know and are able to do on 
a test designed to measure their achievement. The second expectation is an increased emphasis 
on high-quality instruction of the measured content. Although the AA-MAS policy permits a 
reduction in difficulty of the test content and standards following modification, making the test 
easier is not the primary intent of the policy. The primary intent of the AA-MAS policy is to 
improve measurement of abilities on grade-level content for a group of students for whom the 
general state assessment may not yield scores from which valid inferences can be made. The 
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policy permits states to administer the AA-MAS with or without individualized accommodations 
as needed. 

The terms access, accommodations, and modifications all have been used for decades when 
discussing educational testing and the validity of resulting test scores. These terms represent key 
concepts in the world of testing and federal assessment and accountability policies. Thus, these 
terms demand attention to ensure they are understood in the context of emerging issues around 
tests and testing for students with disabilities. 

For educational testing, access is the opportunity for test takers to demonstrate proficiency on 
the target construct of a test (e.g., language arts, mathematics, or science) or a test item (e.g., 
synonyms, homonyms, and homographs). In essence, complete access is manifest when a test- 
taker is able to show the degree to which he or she knows the tested content. Access, therefore, 
must be understood as an interaction between individual test-taker characteristics and features of 
the test itself. 

The purpose of both testing accommodations and modifications is to increase individuals’ 
access to tests. The definitions of these two access-enabling strategies, however, have been the 
subject of debate, in part because of their inconsistent use in the Standards for Educational 
and Psychological Testing (Standards, American Educational Research Association [AERA], 
American Psychological Association [APA], & National Council on Measurement in Education 
[NCME], 1999) and some states’ testing guidelines. Recently, the inclusion of the term modified 
in the AA-MAS has added to the debate about these terms. An examination of the Standards 
finds that the term modification is cited nearly a dozen times in the index. A number of the 
standards refer to a modification as an accommodation. For example, in the Standards’ section 
titled “Testing Individuals with Disabilities,” it is stated, 


The terms accommodation and modification have varying connotations in different subfields. Here 
accommodation is used as the general term for any action taken in response to a determination 
that an individual’s disability requires a departure from established testing protocol. Depending on 
circumstances, such accommodation may include modification of test administration processes or 
modification of test content. No connotation that modification implies a change in construct(s) being 
measured is intended. (AERA et al., 1999, p. 101) 


At this point in time, the implementation of testing accommodations for students with disabil- 
ities is a universally endorsed policy in all states. The modification of test content, however, is 
inconsistent with the definition of a testing accommodation in the majority of state testing accom- 
modation guidelines (Lazarus, Thurlow, Lail, Eisenbraun, & Kato, 2006). Accommodations are 
widely recognized in state testing guidelines as changes to the setting, scheduling, presentation 
format, or response format of an assessment (Kettler & Elliott, in press). Accommodations are 
made to increase the validity of inferences that can be made from a student’s scores, so that those 
scores can be meaningfully compared to scores of students for whom testing accommodations 
are not needed. 

The AA-MAS policy extends the notion of access and the spirit of individualized accommo- 
dations to changes made to item content. Such changes are defined as modifications by most 
test developers and administrators. When item and test alterations are made (e.g., by changing 
the layout, reducing the length of the reading passage, adding graphic support), it is not always 
clear without test results whether the changes affect only access to the test or, in fact, also affect 
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the construct being measured and the subsequent inferences that are drawn from scores. If the 
content of an item or test has been changed and the scores are no longer comparable to the original 
construct to be measured, it has been customary to consider the alteration a modification (Koretz 
& Hamilton, 2006; Phillips & Camara, 2006). To ensure the modifications are acceptable under 
AA-MAS policy, research is needed to confirm the modified test measures the same construct(s) 
as the original test. Indeed, the policy assumes AA-MAS for eligible students measure the same 
grade-level content standards and performance objectives (constructs) as general assessments for 
students without disabilities. It should be noted that some modifications may result in an AA- 
MAS that yields scores that are not comparable to scores obtained from the original test. These 
modifications could still be permissible for an AA-MAS, assuming the content is commensurate 
with the intended grade-level of the test (Kettler, Russell, et al., 2009). 

In summary, the application of the term modification to the context of these recent legislative 
regulations is not acceptable to all parties, and admittedly the use of these terms is subject to 
some controversy. There does not, however, exist a consensus on an alternative term for alterations 
made to item or test content that has been shown to result in tests of unchanged constructs that 
yield scores from which comparable inferences can be made. Given the approach we recommend 
for developing an AA-MAS, we have used the term modification to refer to a process by which 
the test developer starts with a pool of existing test items with known psychometric properties, 
and makes changes to the items, creating a new test with enhanced accessibility for the target 
population. When analyses indicate inferences made from the resulting test scores are valid in- 
dicators of grade-level achievement, the modifications are considered appropriate. Conversely, 
if analytic evidence suggests the inferences made from resulting scores are invalid indicators of 
grade-level achievement, the modifications are inappropriate. Thus, like individualized testing 
accommodations, modifications must be studied to determine their appropriateness. Unlike ac- 
commodations, modifications as conceptualized within AA-MAS policy are intended to afford 
an entire group of students access and better measurement of their achieved knowledge. 


AN EVOLVING MODIFICATION PARADIGM 


Over the past 2 years, we have developed a paradigm for the development and evaluation of 
an AA-MAS, beginning at the level of a single test item. This model has evolved out of two 
federally funded projects—the Consortium for Alternate Assessment Validity and Experimental 
Studies (CAAVES! - Elliott & Compton, 2006-2009) and the Consortium for Modified Alternate 
Assessment Development and Implementation? (Elliott, Rodriguez, Roach, & Kettler, 2007-— 
2010). 

This emerging Modification Paradigm offers a conservative and systematic approach to im- 
proving the accessibility of an item, without changing the difficulty to the point that it would 


1CAAVES was a U.S. Department of Education Enhanced Assessment grant co-directed by Elizabeth Compton and 
Stephen N. Elliott. Several studies on item modification were conducted within this multi-state project during 2007-2009. 
For more information, visit http://peabody. vanderbilt.edu/LSI_Projects/CAAVES-Project-Home.xml. 

2The Consortium for Modified Alternate Assessment Development and Implementation is a U.S. Department of 
Education General Supervision Enhancement grant co-directed by Stephen N. Elliott, Michael C. Rodriguez, Andrew 
T. Roach, and Ryan J. Kettler. Several studies on item modification are being conducted within this multi-state project 
during 2008-2010. For more information, visit http://peabody. vanderbilt.edu/x8314.xml. 
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4. Joan earns $100 per month working part-time in a music store. Look 
at the pie chart that shows how Joan budgets her money each 
month. If Joan sticks to her budget, how much can she spend on 
clothes and entertainment each month? 


Joan's Budget 


Clothes 

20% 
4, Savings 
50% 





Entertainment 
30% 





© A. $50.00 
© B. $40.00 
© Cc. $30.00 
© D. $20.00 


FIGURE 1 A multiple-choice item in original form. 


no longer be an indicator of a grade-level content objective. The first step is to evaluate the 
original item for accessibility. This step is accomplished by appraising the constituent elements 
of the item, attending specifically to barriers that could affect some of the target population. 
In the case of a multiple-choice item from an eighth-grade mathematics test, such as the one 
featured in Figure 1, the elements (or anatomy) of the item that could be modified include the 
item passage or stimulus, any visuals or graphics, the item stem or question, and the response 
options. An evaluation tool for appraising the accessibility of an item across these elements, 
the Test Accessibility and Modification Inventory (TAMI; Beddow, Kettler, & Elliott, 2008) is 
discussed later. Along with the structural inspection of the item, characteristics related to its 
reading load and complexity should be documented. For example, the item in Figure 1 has 42 
words and the Flesh-Kincaid readability index is 4.7. The depth of knowledge of the item, an 
indicator of its complexity, is 2: Skills and Concepts. This depth of knowledge indicates that the 
item involves the application of what the test-taker knows, but that the item has only one correct 
answer (A). Last, item characteristics based on response data should be considered. One type of 
item characteristic that should be considered is the difficulty, operationalized as the percentage 
of persons who responded with the correct answer, which in this case was .69. Rodriguez (2009) 
also suggested considering the proportion of students who selected each of the incorrect answer 
choices, as well as the point biserial correlation between each of the response options and the 
total score to determine how well each response option functions. 

The second step in our modification paradigm is to reduce sources of construct-irrelevant 
variance. Unnecessary language is removed (e.g., “working part-time in a music store” in the 
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4. Pete earns $100 per month. Based on the graph, how much can he 
spend on clothes and entertainment each month? 


Pete’s Budget 


Clothes 


20% 7 






| Savings 
| 50% 


Entertainment 
30% 


© A. $50.00 
© B. $30.00 
© C..... $20.00 


FIGURE 2 A multiple-choice item in modified form. 


example from Figure 1). Irregular names are changed to more easily pronounced names (e.g., 
“Joan” to “Pete”). In the example in Figure 1, the visual became larger and easier to read; the 
portion of the chart that referred to “Savings” was changed to look more like 50%, the proportion 
to which it corresponded. Unnecessary and potentially distracting color was removed from the 
visual. Among the answer choices, white space was increased to help with tracking. Answer 
choice “B” was eliminated on the basis of being the least selected and least plausible distractor 
(i.e., incorrect answer choice). The resulting item is presented in Figure 2. The item is streamlined 
and more accessible. 

The third step in the modification paradigm is to document changes to the item. Completing 
the TAMI on the modified item is one way to document changes to the item. Another helpful 
documentation tool is a log of all of the changes that are made to each item. Also, any changes 
in the aforementioned item characteristics should be mentioned. In our example, the number of 
words was reduced to 20, the Flesh-Kincaid readability index was reduced to 4.4, and the depth 
of knowledge remained 2. Although various stakeholders may disagree on the extent to which the 
word count, readability, and depth of knowledge of an item can be changed and still remain on 
grade level, documentation of these changes is an important step in the modification paradigm. 

The fourth step involves piloting items with students and soliciting their input about the 
items using both cognitive labs and postassessment interviews. Such a step is strongly en- 
couraged by the Standards (i.e., Standard 10.3, “Where feasible, tests that have been modified 
for use with individuals with disabilities should be pilot tested on individuals who have simi- 
lar disabilities to investigate the appropriateness and feasibility of the modifications,” p. 106). 
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Gathering response data during pilot testing allows test developers to identify items with features 
students perceive as confusing. The Standards further suggest information about student re- 
sponse processes and test-taking behaviors can provide evidence to support the construct validity 
of an assessment. In the case of AA-MAS items, student response data can provide impor- 
tant information about the reasons for observed differences in performance across item types 
(original vs. modified) and student groups (students with and without disabilities; AA-MAS 
eligible vs. noneligible students). The use of concurrent think-aloud protocols and follow-up 
questioning may allow researchers to “unpack” unexpected results. This type of evidence has 
been central in the development of an AA-MAS in Arizona, where test items generally in- 
clude features that are intended to reduce or eliminate construct-irrelevant influences on student 
outcomes. 

The final step in the modification paradigm is to field test the new version of the item, ideally 
on a sample of test takers with and without disabilities that also complete the original version 
of the item, and to examine its performance. Item difficulty, discrimination, and reliability are 
all important characteristics to measure. Again, the degree to which these indices are expected 
to change through the modification process may be debatable, but the direction of changes is 
not. We expect modified items to be less difficult, to discriminate ability better, and to have 
better reliability than the original versions of the items. The proportion of test takers selecting 
each response option, as well as the point biserial correlation for each option, should also be 
calculated (Rodriguez, 2009). Documenting changes on these psychometric features is essential. 
This five-step modification paradigm, applied at the item level, is a precursor to a reliability and 
validity argument at the test score level. 


Modifications and Reliability 


The reliability of a set of scores is affected by variance based on the construct being measured, 
or true-score variance, as well as by variance that is based on other factors, or error-score 
variance. Error-score variance can be further subdivided into that which emanates from system- 
atic sources (e.g., test-wiseness, concentration) and that which emanates from random sources 
(e.g., inconsistency in item writing or administration). Appropriate test modifications for stu- 
dents eligible for the AA-MAS should increase test score reliability by decreasing error-score 
variance from random sources. For example, a group of students who are slow readers could 
take a mathematics test primarily consisting of long story problems that were written and admin- 
istered inconsistently (e.g., sometimes using the active tense and sometimes using the passive 
tense, using implausible or absurd answer choices on some items, sometimes arranging ver- 
tically and sometimes horizontally, using visuals that are irrelevant on some items), and their 
scores would reflect a combination of mathematics ability (based on true-score variance), read- 
ing ability (based on systematic error-score variance), and variance that cannot be explained 
(random error-score variance). Modifying these items to be more uniform, and to only contain 
necessary visuals and plausible answer choices, would increase the reliability of the test scores 
by removing some of the random error-score variance. Modifying these items to reduce the 
length and complexity of the reading passages likely would not increase the reliability of the 
test scores. These modifications would improve the validity of the scores, the focus of the next 
section, by reducing systematic error-score variance. All of the modifications in this example 


MODIFYING ACHIEVEMENT TEST ITEMS 535 


would likely make eligible students’ scores more comparable to the scores of students for whom 
the general assessment is appropriate and, as such, would be in the spirit of the AA-MAS 
policy. 

Two methods for estimating reliability, internal consistency and test-retest stability, are often 
used to evaluate achievement tests. Internal consistency, a measure of how well items in a 
set fit together, is commonly estimated using coefficient alpha. Coefficient alpha is related to 
interitem correlations, and it tends to be a high estimate of reliability. Coefficient alpha is not 
sensitive to systematic error based on maturation or practice between administrations, because 
it is calculated based on a single administration. Test-retest stability is another indicator of the 
reliability of a test and is typically calculated by having students complete the same or equivalent 
forms of a test at different points in time, and then calculating the correlation between the two 
sets of test scores. A key difference between test-retest stability and internal consistency is 
that test-retest stability is based on comparisons between test scores, rather than an average of 
comparisons among items and a test score. Test—retest stability is sensitive to maturation between 
the two times of measurement, and correlations typically are expected to be lower when intervals 
between testing events are longer. Test-retest designs that utilize the exact same test on both 
occasions are more sensitive to practice effects, whereas designs that utilize parallel forms are 
less sensitive to practice effects but are sensitive to any differences that may exist between the 
two forms. 

Modifications made to items and tests for the AA-MAS are intended to preserve or increase 
reliability for students who would be eligible. Such improvements should be observable whether 
reliability is characterized as coefficient alpha or test-retest stability. Measurement of reliability 
is a critical first step toward establishing a validity argument for both unmodified and modified 
tests. 


Modifications and Validity 


The validity evidence for inferences drawn from AA-MAS scores is central to determining 
whether the instrument has value for its intended purpose (Kettler, Russell, et al., 2009). Validity 
is the degree to which scores from an instrument reflect the construct that they are intended to 
measure. Since the 1985 Test Standards (APA, 1985), a unified view of validity has prevailed, 
replacing a previous emphasis on construct, content, and criterion validity. This view of validity 
has been expanded to embrace an evidence evaluation orientation in the most recent version of 
the Standards (AERA, APA, & NCME, 1999). This approach to validity values evidence about 
the test content, response processes, relationships with other variables, internal structure, and 
consequences of testing. 

Evidence based on content validity is the degree to which items or tests appear to represent 
the content that they are designed to measure. Content validity can be prioritized during the 
modification process by having experts involved and can be evaluated by having experts rate items 
for importance, making comparisons with the grade-level content standards that are supposed to 
be measured. Content experts can also evaluate items and tests in their original conditions, and 
make comparisons with the modified conditions, to determine if any change in content validity 
is caused by the modification process. 
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Validity evidence based on response processes refers to whether the students responding to an 
item or test interpret it in the intended way. One promising method for measuring and documenting 
validity evidence based on responses has been through cognitive lab methodology. Cognitive 
labs (also often called cognitive interviews, verbal protocols, or think-aloud studies) involve 
participants verbalizing their thoughts and strategies while completing tests. The application of 
cognitive lab methods to item and test modifications is discussed in detail later. Other methods 
for measuring validity based on responses include item analysis protocols, which would prompt 
students to list their specific considerations evoked by an item, and surveys about the testing 
experience. Both of these methods would involve prompting students after they had completed 
items, rather than during item completion. Qualitative analysis would then be used to determine 
whether students taking modified tests reported equal or less cognitive effort, compared to original 
conditions. 

Validity evidence based on internal structure is the degree to which the different parts of an 
assessment instrument fit together as intended. This process often involves evaluating items and 
their relationships with each other. Factor analysis is performed to evaluate internal structure 
validity, indicating whether the data-based pattern of relationships among items matches the 
intended theoretical model of scales and subscales. Item response theory is also used to determine 
the match between items and theory, including whether the items are distributed in terms of 
difficulty in the way that the developers or modifiers intended. Both factor analysis and item 
response theory can be used to evaluate a finished product, or as part of an iterative development 
or modification process. Research comparing original with modified conditions of items and 
tests would determine whether the factor structure remained unchanged, and whether the items’ 
difficulties remained on grade level. 

Validity evidence based on relationships with other variables is the degree to which scores 
from the newly developed or modified test converge with scores from measures of similar con- 
structs but diverge with scores from dissimilar constructs. New or modified tests can be evaluated 
by measuring the correlation of scores from the test being validated with scores from a test 
with established properties. Validity based on relationships with other variables can also be 
measured by evaluating the performance of known groups (e.g., eligible students and noneli- 
gible students), to determine whether the groups differ in the expected direction. In this case, 
group membership becomes the other variable, against which scores from the modified items 
or tests can be evaluated. A special case of validity evidence based on relationships with other 
variables, the interaction paradigm, has been proposed as a framework for evaluating improved 
access based on accommodations or modifications (Kettler, Rodriguez, et al., 2009; Phillips, 
1994). The interaction paradigm indicates that the gap between scores from eligible students 
and scores from noneligible students should be reduced through the modification process, be- 
cause students who would be eligible should gain greater access as a result of the modification 
process. 

All of these types of validity evidence can be used to evaluate a modified test as well as to 
compare a modified test with the original test. A comprehensive argument for the validity of an 
AA-MAS should be based on a combination of evidence from these sources. The goal is that 
measurement of achievement for grade-level content for students eligible for an AA-MAS should 
become more reliable, valid, and comparable to measurement of the same content for students 
who are not eligible and who complete the general assessment. The challenges and consequences 
of pursuing this goal are discussed in the next section. 
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Consequences of AA-MAS 


A fifth form of validity evidence mentioned in the Standards, consequential validity, is relevant 
only to the relationship between test scores and the constructs that they are designed to reflect, 
to the extent that error-score variance can lead to negative consequences from test scores. Other 
consequences of testing (positive and negative) are also important to evaluate but are not directly 
related to validity. Evidence about the consequences of an AA-MAS and of other types of testing 
is difficult to evaluate in the short term and depends on gathering longitudinal data on a wide 
variety of variables. The U.S. Department of Education (2007a) identified intended consequences 
of implementation of the AA-MAS, including (a) providing students with disabilities access to 
grade-level content, (b) ensuring that the performance and progress of students with disabilities 
are accurately assessed, and (c) allowing teachers and schools to get credit for this progress and 
performance. 

Elliott, Kettler, and Roach (2008) identified several potential unintended consequences, in- 
cluding (a) that there may be time and money for test modification and professional develop- 
ment, (b) many accommodations typically used with these students may not be necessary on 
the modified tests, and (c) lowered achievement standards may not lead to improvements in 
motivation or learning. This latter consequence is particularly complicated given the empha- 
sis in the AA-MAS policy on allowing standards to be easier, so long as they are not below 
grade level. The term grade level is not operationally defined, notwithstanding the legitimate 
concern that excessively lowering achievement standards could undermine students’ motiva- 
tion to learn. Another consequence of the AA-MAS policy is that modifications may yield 
scores that are not comparable to those from the regular assessment. This could occur even 
if both assessments measure the same grade-level content, because the proficiency level de- 
scriptors on which cut scores for the AA-MAS are set may not match the descriptors used 
for the general assessment (see Egan et al., 2009/this issue, for a thorough discussion of this 
topic). One final unintended consequence is the mathematical likelihood that percentages of 
students who achieve proficiency on the general assessments will increase, because students 
who consistently achieved below proficient are removed from the population completing these 
tests. 

These intended and unintended consequences are representative of the manifold issues and 
concerns that can only be evaluated by tracking trends in the data over the course of several years, 
both preceding and following the implementation of AA-MAS. Until substantial research and 
evaluation data can be collected to answer these questions, we will rely more on theory to guide 
our practice of item and test modification. 


TEST ACCESSIBILITY: THEORY AND RESEARCH 


Test accessibility is the extent to which a test (and its constituent item set) permits the entirety 
of the test-taker population to demonstrate proficiency on the target construct(s) of the test. To 
the extent a test demands physical, material, or cognitive resources in excess of those required to 
measure the intended construct, inferences made from the scores on the test likely reflect in some 
part the test-taker’s incomplete access to the test. 
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Current federal legislation requires the application of ‘universal design principles to the devel- 
opment of all statewide and district-wide achievement tests. Universal design, as defined in the 
Assistive Technology Act (P.L. 105-394, 1998), is 


a concept or philosophy for designing and delivering products and services that are usable by people 
with the widest possible range of functional capabilities, which include products and services that 
are directly usable (without requiring assistive technologies) and products and services that are made 
usable with assistive technologies. (§3(17)) 


Although the term accessibility is not used in this definition, universal design principles as 
applied to assessment technology clearly are intended to address issues of access while responding 
to the concern raised in the Standards that the use of individualized accommodations may increase 
measurement error. The legislation provides the rationale for the use of universal design principles 
as follows: 


The use of universal design principles reduces the need for many specific kinds of assistive tech- 
nology devices and assistive technology services by building in accommodations for individuals 
with disabilities before rather than after production. The use of universal design principles also in- 
creases the likelihood that products (including services) will be compatible with existing assistive 
technologies. These principles are increasingly important to enhance access [italics added] to infor- 
mation technology, telecommunications, transportation, physical structures, and consumer products. 
((PL105-394(§3(10)) 


In anticipation of aforementioned regulations under No Child Left Behind, permitting states 
to develop AA-MAS, the CAAVES project team conducted a series of studies to examine the 
differential effects of item modifications on the performance of students across disability and 
eligibility groups. To assist in the item modification process with the goal of developing more 
accessible tests, Beddow et al. (2008) developed the TAMI. The TAMI was designed to assist 
with the process by which general assessment items are modified in response to the demand 
for universally designed assessments to be used for the AA-MAS. Thus, the instrument has two 
purposes: The first is to meaningfully quantify the construct of accessibility with regard to tests 
and items; the second is for use as a training tool to guide item writers in the development 
and modification of tests and items that are highly accessible. To wit, the development of the 
TAMI was influenced by (a) universal design principles for learning and assessment, (b) cognitive 
load theory, (c) research on test and item development, and (d) guidance on web and computer 
accessibility. 


Universal Design for Learning and Assessment 


The Center for Universal Design (1997) lists seven primary aspects of universal design: (a) 
equitable use, (b) flexibility in use, (c) simple and intuitive use, (d) perceptible information, (€) 
tolerance for error, (f) low physical effort, and (g) size and space for approach and use. The last 
decade has seen a shift in focus toward applying universal design principles across the educational 
arena, called universal design for learning (e.g., see Rose & Meyer, 2006) and universal design for 
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assessment. Several documents from the National Center on Educational Outcomes informed the 
development of the TAMI, specifically with respect to universal design for assessment principles 
(e.g., Johnstone, Thurlow, Moore, & Altman, 2006; Thompson, Johnstone, & Thurlow, 2002). 


Cognitive Load Theory 


As conceptualized by Sweller and his colleagues, cognitive load theory (CLT; Chandler & 
Sweller, 1991) is a logical extension of Miller’s (1956) work on processing limitations and 
working memory, exemplified in Miller’s now famous “Magical Number Seven, Plus or Minus 
Two” article. Based on an information-processing framework, CLT has heretofore been conceived 
as a model for understanding the demands of learning tasks and is grounded in three primary 
assumptions about how the mind works: (a) the mind processes information through two separate 
channels: the auditory/verbal channel and the visual channel, (b) each of these channels has a 
limited capacity for processing information, and (c) both channels are required for most learning 
tasks. Proponents of CLT posit that to properly gain knowledge from instruction, students must 
(a) attend to the presented material, (b) mentally organize the material into a coherent structure, 
and (c) integrate the material with existing knowledge. Thus, the efficiency of instructional tasks 
depends on the extent to which element interactivity and the requisite cognitive resources needed 
to process essential information are minimized. 

Accordingly, CLT disaggregates the cognitive demands of learning tasks into three load types: 
intrinsic load, germane load, and extraneous load. Intrinsic load refers to the amount of mental 
processing that is requisite for completing a task. Germane load refers to cognitive demands that 
are not necessary for gaining essential knowledge but enhance learning by facilitating gener- 
alization or automation (e.g., lessons that require learners to extend learned concepts to arenas 
outside the classroom or apply them to novel situations). Extraneous load refers to the demand for 
cognitive resources to attend to and integrate nonessential elements that are preliminary to actual 
learning, but are nonetheless required for a learning task. Proponents of CLT argue that learning 
tasks should be designed with the goal of minimizing the demand for cognitive resources that 
are extrinsic to the goals of instruction. The triune model of cognitive load was encapsulated by 
Paas, Renkl, and Sweller (2003): “Intrinsic, extraneous, and germane cognitive loads are additive 
in that, together, the total load cannot exceed the working memory resources available if learning 
is to occur” (p. 2). 

Notwithstanding the broad overlap between instruction and testing, CLT heretofore has had 
little research application to school-age students with or without special needs or to the assessment 
of student learning. Considering the numerous similarities between instructional tasks and the 
variety of tasks required in many forms of tests, the development of the TAMI (Beddow et al., 
2008) focused explicitly on the degree to which cognitive load demands may impact a test- 
taker’s ability to demonstrate performance on assessments. Particular attention was paid to how 
CLT has been used to understand the cognitive demands of multimedia learning. To the extent 
the cognitive demands of an assessment are intrinsic to the target constructs of the assessment, 
inferences made from test results are likely to represent the person’s actual competence on the 
constructs. Extraneous load demands by an assessment item interfere with the test taker’s capacity 
to demonstrate performance on the target construct and should be eliminated from the assessment 
process. 
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Further, germane load, although enhancing learning at the instructional level, should be con- 
sidered for elimination as well. Unless an assessment task has the dual purpose of both instruction 
and assessment, the items on a test should demand only those cognitive resources intrinsic to the 
target constructs they are intended to measure. The reduction of the germane load to an assess- 
ment task may represent a decrease in the depth of knowledge of an item, if it requires additional 
elements or interactivity among elements. It should be noted that the ability of the test taker to 
integrate elements to complete a task depends not only on the materials themselves but also on 
his or her expertise. Thus, the decision to include or exclude germane load from assessment tasks 
should be made deliberately and with knowledge of the population being tested. 

Clark, Nguyen, and Sweller (2006) synthesized the CLT research and generated a set of 29 
guidelines for maximizing efficiency in learning. The majority of the recommendations focus on 
reducing redundancy, eliminating nonessential information from text and visuals, and integrating 
information from dual sources. There are also a number of cautionary considerations when using 
audio to supplement instruction. Where applicable, these guidelines were incorporated into the 
TAMI. 


Research on Test and Item Development 


The TAMI also was influenced by the collective expertise of test and item development re- 
search from a number of scholars. Specifically, the instrument includes adaptations of several 
recommendations from a taxonomy for writing multiple-choice items by Haladyna, Downing, 
and Rodriguez (2002) based on a comprehensive review of research. The authors recommend 
avoiding overspecific and overgeneral content in items, avoiding trick items, minimizing the use 
of negative stems (e.g., using the words not and except), and keeping vocabulary as simple as 
possible given the population being tested. Further, the authors suggest the reading load of items 
should be limited to that which is necessary to measure the given construct. 

A large number of Haladyna et al.’s (2002) guidelines deal with the construction of the 
answer choices (i.e., the key and distractors). Specifically, to reduce the potential for cueing, they 
recommend placing choices in logical or numeric order, keeping choices homogeneous in terms 
of content and/or grammatical structure, keeping the length of choices about equal, ensuring all 
distractors are plausible (avoiding specific determiners such as always and never, and excluding 
absurd options), and balancing the answer choices in terms of content. 

Based on a meta-analysis of more than 80 years of research on item development, Rodriguez 
(2005) concluded that three answer choices are optimal for multiple-choice items. Increasing the 
number of plausible distractors beyond two does little or nothing to improve item characteristics 
(i.e., difficulty, discrimination, and reliability) and adds to the cumulative reading load of a 
test. Accordingly, the Answer Choices category descriptors on the TAMI implicitly support the 
practice of reducing the number of response options from 4 to 3. It should be noted that when 
reducing the number of distractors from 4 to 3, Rodriguez recommends eliminating the least- 
selected distractor unless an implausible distractor would remain. Although eliminating a more 
plausible distractor likely would decrease the difficulty of an item, the effect may be due to 
cueing of the correct response by an implausible distractor, theoretically reducing the validity of 
a subsequent inference about the meaning of a correct response as it regards proficiency on the 
target construct. 
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TABLE 1 
Theory-Based and Research-Supported Modifications 





Modifications to reduce unnecessary language load 

Rewrite to replace pronouns with proper nouns 

Simplify sentence and text structure with an emphasis on clarity 

Reduce vocabulary load and non-construct subject area language 

Chunk and segment the text into manageable pieces 

Base the item on the content it is written to measure by removing any trivial content 

Minimize the amount of reading necessary by reducing excess text 

Replace negatives (e.g., NOT or EXCEPT) with positive wording 

Edit the items for errors in grammar, punctuation, capitalization, and spelling 
Modifications to answer choices 

Eliminate any implausible distractors until as few as three choices remain 

Move a central idea that is in the item choices to the item stem 

Avoid cuing for a correct or incorrect answer 

Place answer choices in a logical order and make them structurally homogenous 
Other general modifications 

Make items more factual, rather than opinion-based 

Add white space to make tracking easier 

Remove visuals that are not necessary or helpful 

Format items to be read vertically 

Use bold text for important words 





Although Haladyna et al.’s (2002) and Rodriguez’s (2005) research preceded the AA-MAS 
policy, many of their findings are relevant to the modification process. More recently, researchers 
have inventoried theory-based and research-supported modifications, as helpful guides to the 
modification process (Hess, McDivitt, & Fincher, 2008; Kettler, Russell, et al., 2009). Table 1 
contains a list of these practices to consider when modifying items. 


Guidance on Web and Computer Accessibility 


In Bennett’s (2001) “How the Internet Will Make Large-Scale Assessment Reinvent Itself,” he 
argued the advancing pervasiveness of computer technology into all areas of modern life would 
lead, for better or worse, to the inevitable subsumption of standardized testing. Indeed, the 
ever-increasing use of online testing across the range of student assessment types supports his 
hypothesis. Commensurate with this apparent trend, the TAMI (Beddow et al., 2008) includes a 
Computer-Based Test Analysis section to be used by developers of online tests to ensure adequate 
attention is paid to the accessibility of these tests for as many students as possible. In addition, 
the authors reviewed documents by the Web Accessibility Initiative (W3; 2008), which has 
proposed a set of guidelines to ensure that Web content is accessible to all users, including those 
with disabilities. According to W3, the Web Content Accessibility Guidelines are expected to 
become a Web standard recommendation by 2009. The guidelines focus on four key principles: 
perceivable, operable, understandable, and robust (for an example of a universally designed 
test delivery system that integrates many of the recommendations included in the TAMI, see 
NimbleTools at http://www.nimbletools.com). 
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The existing version of the TAMI is divided into two sections: Item Analysis and Computer- 
Based Test Analysis. The Item Analysis section is divided into categories based on the anatomy 
of an item. The Computer-Based Test Analysis section is divided into categories based on the key 
elements of a multimedia test system. Each section of the TAMI corresponds to an Accessibility 
worksheet for documenting ratings and suggested changes. The rater reads the descriptors in each 
category and records accessibility ratings for the categories on the worksheet using a 4-point 
scale, ranging (0 = not accessible, 1 = minimally accessible, 2 = moderately accessible, 3 = 
maximally accessible), yielding a summative total accessibility rating. Research on the utility of 
the TAMI is ongoing, but early field studies have indicated test developers and educators find the 
instrument useful to guide the development of accessible tests. Some of this research has involved 
the use of the TAMI to modify items for use in cognitive labs, where students provide feedback 
relevant to universal design principles, CLT, and item development. 


Using Cognitive Labs to Guide Item and Test Modifications 


The value of information regarding student responses and perceptions in supporting the develop- 
ment of assessments is addressed at multiple points in the Standards (AERA, APA, & NCME, 
1999). Standard 10.3 indicates, “Where feasible, tests that have been modified for use with in- 
dividuals with disabilities should be pilot tested on individuals who have similar disabilities to 
investigate the appropriateness and feasibility of the modifications” (p. 106). The Standards also 
suggest information regarding student response processes can provide evidence to support the 
construct validity of an assessment. “Questioning test takers about their performance strategies 
can yield evidence that enriches the definition of a construct ...” (AERA, APA, & NCME, 
1999, p. 12). The Standards further note the potential contribution of student response data: “Pro- 
cess studies involving examinees from different subgroups can assist in determining the extent 
to which capabilities irrelevant or ancillary to the construct may be differentially influencing 
(student) performance” (p. 12). We have found that student response data can provide important 
information about the reasons for observed differences in performance across item types (original 
vs. modified) and student groups (students with and without disabilities; AA-MAS eligible vs. 
noneligible students). The use of concurrent think-aloud protocols and follow-up questioning 
allows researchers to “unpack” unexpected results. For example, differential item functioning 
may indicate a particular item was difficult for students with identified disabilities in comparison 
to their peers. Recording students’ concurrent verbalizations while solving the item in question, 
as well as questioning students following completion of the task, may illuminate item features 
that contribute to the observed results. This type of evidence is central in the development of an 
AA-MAS, where test items generally include features that are intended to reduce or eliminate 
construct-irrelevant influences on student outcomes. However, with the exception of a few studies 
on testing accommodations (e.g., Elliott & Marquart, 2004; Fulk & Smith, 1995; Kosciolek & 
Ysseldyke, 2000; Lang, Elliott, Bolt, & Kratochwill, 2008; McKevitt & Elliott, 2003), the use of 
student response data for the purpose of test construction or test validity is virtually absent in the 
research literature. 

In their seminal book Protocol Analysis: Verbal Reports as Data, Ericsson and Simon (1993) 
describe the rationale for the development of think-aloud methods for obtaining concurrent and 
retrospective verbal reports. Drawing on recent research on information processing, Ericsson 
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and Simon developed an approach to collecting concurrent and retrospective verbal reports that 
demonstrated minimal influence on participants’ problem solving and cognitions. Because of 
their desire to create “hard data” about individuals’ cognitive processes, Ericsson and Simon’s 
approach is somewhat restrictive. For example, the experimenter provides limited prompting or 
encouragement, and often sits behind the participant to discourage interaction. Moreover, “it is 
important that subjects verbalizing their thoughts while performing a task do not describe or ex- 
plain what they are doing—they simply verbalize the information they attend to while generating 
the answer” (p. xiii). Ericsson and Simon (1993) suggested that think-aloud verbalization dur- 
ing assessment tasks reflects the cognitions’ simultaneous happening in participants’ short-term 
memory. Students may find it difficult to verbalize their problem solving on test items that are too 
simple and involve skills and concepts at a level of automaticity (i.e., stored in long-term mem- 
ory). Conversely, items that are too difficult and complex may result in confusion and frustration 
on the part of student respondents. 


Components of a Cognitive Lab: The Arizona Example. In our efforts to develop AA- 
MAS, we collected data from students (with and without disabilities) taking short tests as part 
of a think-aloud lab. The work of Johnstone, Bottsford-Miller, and Thompson (2006) influenced 
the design of our think-aloud method. This method has become a common part of our test 
development procedures and is useful for evaluating salient characteristics of test items and 
testing procedures. The think-aloud procedure typically involves two steps: 


Step 1. During video- and audio-recorded sessions, data are collected “in real time” by 
asking students to think-aloud as they complete or solve assessment tasks. Researchers verbally 
prompt students as infrequently as possible to avoid distracting them during problem-solving 
activities (Ericsson & Simon, 1993). If students are silent for several seconds, researchers can use 
neutral prompts, such as encouraging the student to “keep talking” (Johnstone, Bottsford-Miller, 
et al., 2006). 


Step 2. Once the think-aloud process is complete, researchers may ask follow-up questions 
about the process. Answers to these questions can supplement any unclear data derived from the 
students’ verbal responses (e.g., which item modifications did students find most useful). These 
follow-up questions may also be useful for probing the understanding of students who, as a result 
of skill deficits or disability, are unable to meet the cognitive demands of thinking aloud while 
problem solving (Johnstone, Bottsford-Miller, et al., 2006). 

Cognitive labs generally are conducted with rather small numbers of participants to allow for 
in-depth analyses of how individuals process and use information. Although no comprehensive 
review articles have been published on cognitive labs, the studies that have been completed 
generally feature fewer than a dozen participants and use direct observations, follow-up questions, 
and post hoc reviews of permanent products (Johnstone, Bottsford-Miller, et al., 2006; Roach, 
Beddow, Kurz, Kettler, & Elliott, 2009). Cognitive labs with small, representative samples of 
participants provide some advantages for in-depth examinations of responses; however, small 
samples of participants also create limitations for data analyses and generalization of the results. 
Cognitive labs can play an important role in the early phases of the item and test development 
process when developers are exploring the function and utility of new materials or item formats. 
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In our collaborative work with the Arizona Department of Education to build an AA-MAS, we 
used a cognitive lab to address a number of questions about test items and the students potentially 
eligible to take such an assessment. Specifically, we asked, 


1. Do students eligible for an AA-MAS perform better on modified grade-level test items than 
on original test items? If so, is this improvement greater than that experienced by noneligible 
students? 

2. How do students view the modified versus original grade-level test items? What test/item 
features, if any, did they prefer or perceive as helpful? 

3. What is the relationship between classroom instruction on an objective and student perfor- 
mance on an item designed to measure that objective? 

4. Are there significant qualitative differences between the test-taking behaviors of eligible 
students versus noneligible students? 


The main materials used in this Arizona study with students were grade-specific test booklets 
composed of 12 items (three original and three modified in both reading and mathematics). The 
original and modified items were clustered in subsets and the order of the original and modified 
item subsets was counterbalanced across Forms A and B at each grade level. 

At least one of the teachers for each participating student was asked to complete a Curricular 
Experiences Survey (see Figure 3). This survey listed the content standard objective for each 
item tested and asked teachers to report the amount of instructional coverage provided for each 
objective. Thus, the purpose of the survey was to document the degree to which students had 
been provided an opportunity to learn the content assessed by test items. 

Testing of students occurred at school sites in each of the school districts. Testing sessions 
ranged from approximately 35 min for noneligible students to more than 60 min for many of 
the eligible students. Investigators worked in pairs with each student; one of the investigators 
prompted the student during the think-aloud process and administered the follow-up survey, 
whereas the other investigator ensured the integrity of the video- and audio-tapings. Following 
the directions suggested by Johnstone, Bottsford-Miller, et al. (2006), an investigator explained 
the think-aloud procedures and modeled the process of verbalizing while thinking. The script 
used to explain the process is presented as Figure 4. 

After explaining instructions and providing a short demonstration of how to verbalize, the 
investigator asked the student to engage in a couple of sample items to practice verbalizing his or 
her thoughts. Following this, the testing started with every student completing reading items and 
then mathematics items. Following Johnstone, Bottsford-Miller, et al.’s (2006) recommendation, 
investigators prompted students only when they were silent for 10 consecutive sec. If students 
verbalized infrequently while working on test items, they were reminded to “keep thinking aloud” 
or “keep talking.” Other than these prompts, the investigators remained silent while students were 
thinking aloud to avoid disrupting their thought patterns. 

After each test (reading and mathematics) was completed, the student was asked to rate the 
perceived difficulty of each item he or she had completed (see Figure 5). Student responses to 
these questions provided one form of “cognitive effort” or conversely “cognitive ease” data for 
calculating efficiency metrics (as developed by Clark et al., 2006). We also recorded the amount 
of time students spent completing each test item as another measure of “cognitive effort.” Both 
measures of “cognitive effort” ultimately were combined with data on student performance to 


MODIFYING ACHIEVEMENT TEST ITEMS 545 


Student’s Name: 


Directions: For the student listed above, indicate the extent of content coverage (# of lessons provided during this 
current school year) for each of the following objectives. 


NC = No coverage (Not an objective addressed during this school year) 

P= Plan to cover this objective later in this school year 

1 = Minimal coverage (1-5 lessons to this point in the school year) 

2 = Moderate coverage (6 -10 lessons to this point in the school year) 

3 = Extensive coverage (11 or more lessons to this point in the school year) 

4 = Intensive, systematic coverage (daily/neatly daily instruction throughout the school year) 


bibs Of the SRATES project ——s Objective Rae 
the relevant meaning of a word. 
and ppo Hn g details. 

3. Describing the historical and 
cultural aspects found im cross- 
culture works. 

4 Determining the author's 

maim purpose. 





FIGURE 3 Excerpt from the Curricular Experiences Survey. 


determine item modification effects on efficiency as defined by Clark et al. At the end of each 
content area subtest, students were asked follow-up questions to (a) clarify any inconsistencies or 
confusion regarding their think-aloud responses, and (b) gather additional information about their 
perceptions of the original and modified test items. Student responses to these follow-up questions 
also were recorded. After completing the cognitive lab session, the investigators administered a 
curriculum-based, grade-level oral reading fluency prompt to each student. 

In our Arizona Cognitive Lab Study, we expected students without disabilities would correctly 
answer a higher percentage of all items than their classmates with disabilities. In particular, 
however, we expected both groups of students would (a) perform better on the modified items 
than the original Arizona’s Instrument to Measure Standards items, (b) indicate the modified items 
required less effort to answer than their original sibling items, and (c) perceive the modifications 
to items to be helpful or make items easier to answer. We collected evidence about the items 
(e.g., depth of knowledge, readability level, difficulty) and students’ performances, including 
percentage of items correct, perceptions of effort needed to answer an item, and reading fluency. 
Collectively, the evidence was interpreted to support the first two expectations. With regard to 
the third expectation, the trends in the evidence indicated that modified items were generally 
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“We are interested in how students solve items on tests, so we want to ask you and other 
students to answer some test problems for us and let us listen to how you do that. We are 
not as interested in the answer you come up with as we are with how you are thinking and 
what you are thinking about while you’re answering the test questions. If for any reason 
you would like to stop or take a break at any time, please ask to stop or to take a break.” 


The researcher will strive to be general and honest about our interests in student responses and 
respectful of the contribution each student can make to the research process. 


The student should not feel that he/ she is being judged or that he/she needs to perform ina 
certain way or to a specific standard. Any extraordinary pressure may affect his/her behavior and 
introduces a bias. 





The researcher will ask the student to “parrot” back what he or she was told about today’s 
session by the recruiting person or teacher. If the researcher finds that the student has been given 
information that is biasing or confusing that might affect the session, he or she will re-explain the 
process in order to rectify student understanding. 













“What were you told we were going to do today?” 


The researcher will demonstrate interest about what the student does and why he/she does it. 
Also the researcher should tell the student that we will be videotaping the session and let him/her 
know when the camera is turned on. 





“What you say is really important, so we are going to run this camera to make sure that we 
don’t forget anything.” 


Provide practice task: Each student will be given (at least one) practice task to familiarize him or 
her with thinking aloud while working through an item. First, the researcher will answer an item 
and then ask the student to answer one. (The camera is not turned on for the practice.) The 
researcher will give the following instruction: 






“I’m going to think out loud while I answer this problem. That means I’m going to say 
everything that goes through my mind.” (Researcher completes a practice item while 
thinking out loud.) 






“Now I’m going to ask you to answer an item the same way. Just say everything that goes 
through your mind while you figure out the item.” (Student completes a practice item(s) 

while thinking out loud until they can clearly demonstrate the process before proceeding to 
the research items.) 






“Remember, | am not as interested in the answer to the item as much as how you are 
thinking about the task. Do you have any questions about what we just did?” 





FIGURE 4 Think-aloud protocol script. 


How hard did you have to work to answer the reading/math item above? 
1 2 3 4 S 6 7 
Not Very Very 
Hard Hard 


FIGURE 5 Sample Cognitive Effort/Cognitive Ease question. 
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perceived as helpful or similar to original items. In particular, modifications did no harm and may 
even be helpful from the students’ perspectives. 

Making generalizable conclusions from data based on relatively small samples of students and 
collected at one point in time during a cognitive lab must be done cautiously but can facilitate the 
next steps in the test development process. In the case of Arizona, the results of the cognitive lab 
lead to the general endorsement of a collection of item modification tactics (bolding, embedding 
of visuals and questions, distractor elimination, and language simplification) for reading items 
and the further refinement of some mathematics items where these tactics could be used more 
consistently. 


Experimental Research on Item Modification 


As part of the CAAVES project, one large scale (n = 756) study has been completed examining 
the effects of packages of modifications (often including language simplification, adding graphic 
support, bolding key words, and eliminating one distractor) on the performance of three groups 
of eighth-grade students across conditions in reading and mathematics. A rubric with wording 
taken directly from the guidelines, coupled with the presence or absence of an Individualized 
Education Program, was used to define the three groups: students with disabilities who would be 
eligible for an AA-MAS (SWD-Es), students with disabilities who would not be eligible for an 
AA-MAS (SWD-NEs), and students without disabilities (SWODs). Every student in the study 
completed items in an original condition, in a modified condition, and in a modified condition 
with limited reading support. The reading support condition was extensive in mathematics but 
was limited to the directions and the item stem or question in reading. 

Main findings of the CAAVES experimental study included that (a) the reliability of forms 
could be preserved across modification conditions, (b) modification improved the performance 
of all three groups of students, and (c) eligible students experienced a differential boost in mean 
item difficulty when ability was controlled within a Rasch model (Elliott et al., 2009; Kettler, 
Rodriguez, et al., 2009). Across the two content areas (reading and mathematics), three groups, 
and three modification conditions, coefficient alphas ranged from .85 to .94 based on 39-item 
forms. No meaningful differences were found based on group status or condition, indicating that 
the modification process did not substantially decrease reliability. As would be expected, across 
conditions, SWODs scored the highest. Also across conditions, SWD-NEs scored higher than 
did SWD-Es (Elliott et al., 2009). Students with disabilities who would be eligible for AA-MAS 
exhibited slightly higher effect sizes (Cohen’s d for reading = .40, mathematics = .26) than did 
SWD-NESs (reading = .38, mathematics = .21) or SWODs (reading = .37, mathematics = .15). An 
interaction paradigm, which was not apparent when characterizing performance based on mean 
scores, was present when characterizing performance in terms of mean item difficulty. When 
using a Rasch model to equate the ability levels of the three groups, the impact of modifications 
on item difficulty was significantly larger for SWD-Es than for the other two groups, indicating 
that the modifications were especially helpful for providing access to students who would be 
eligible (Kettler, Rodriguez, et al., 2009). 

Exploratory analyses conducted as part of the CAAVES experimental study addressed which 
modifications were included in particularly effective packages, and which were included in 
packages that were not particularly effective. Nine particularly effective packages were identified 
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as having resulted in an interaction between group and condition, reducing the gap between 
eligible and noneligible students. Five particularly ineffective packages were also identified, 
based on items increasing this gap. These analyses revealed that shortening the stems of items 
was an effective modification across content areas, and that adding graphics to reading items 
was a particularly ineffective modification. No pattern was found for the modification of adding 
graphics to mathematics items. The elimination of a distractor and the increase of white space 
could not be evaluated in these analyses, because they were a part of all packages, effective 
and ineffective. In a posttest survey of this sample, Roach et al. (2009) found that students 
from all groups reported that items were equally difficult across conditions, but that SWD-Es 
and SWD-NEs were more likely to find items toward the beginning of the test easier, perhaps 
indicating perceived fatigue toward the end of the test. With regard to specific modifications, 
students endorsed the use of bold font for key terms, as well as the removal of a distractor. 


SUMMARY AND CONCLUSIONS 
Precautions to Modifying Items and Tests 


The research that has been done so far has raised a number of precautions to modifying items 
and tests for an AA-MAS. One modification that is often considered is allowing portions of a 
test to be read aloud to students, either by a proctor or by computer voiceover. In the CAAVES 
experimental study, the modified condition that featured reading support yielded even larger effect 
sizes than did the modified condition without reading support, when both were compared with the 
original condition. These effects sizes were very similar for SWD-Es (Cohen’s d for reading = .50, 
mathematics = .31), SWD-NEs (reading = .49, mathematics = .25), and SWODs (reading = .38, 
mathematics = .20). Although a reading support modification is generally considered acceptable 
in content areas other than reading, states vary greatly in their interpretation of whether and to 
what degree reading support can be used as a modification on a reading test. The concern with 
providing read aloud support to passages on reading items is that the modification will go beyond 
reducing just content irrelevant variance, to also reducing content relevant variance. In other 
words, some of the meaningful information about students’ abilities to read would be lost. This is 
not a concern when considering a mathematics or science test, for which reading is an access skill. 
However, before eliminating the possibility of any reading support on a reading test, one should 
consider that the sample of students who would be eligible for an AA-MAS are typically very 
slow readers. Students in our Arizona Cognitive Lab study who were in the noneligible group 
typically read about twice as many words per minute, compared to students from the eligible 
group, regardless of grade level. This characteristic would appropriately affect their performance 
on items designed to measure fluency. However, on a broader reading test that includes items 
designed to indicate comprehension and vocabulary, fluency is an access skill. The decision about 
providing reading support in these cases is not as clear, as differences in fluency are likely to 
affect students’ abilities to show what they know and can do in other areas of reading. 

A couple of precautions about the practice of eliminating one distractor from an item must be 
considered. One precaution that was previously mentioned is that properties of multiple-choice 
items and tests are best preserved when the least plausible distractor is eliminated (Rodriguez, 
2005). Eliminating a distractor that is chosen a high percentage of the time is likely to affect more 
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than just the reading load of the item, and previous characteristics such as reliability, difficulty, 
and discrimination will likely be very different for the modified version. When considering the 
distractor that is least plausible, it is helpful to have item statistics that are calculated based on the 
population for whom the test is intended (Abedi, Leon, & Kao, 2007). Another precaution related 
to distractor elimination is that content validity may be changed in the process. In some cases, 
each of the answer choices may contribute to the content standard to which the item was written, 
and removal of a distractor might change alignment. One final precaution related to distractor 
elimination is that three answer choices is not the optimal number in all cases. Some items are 
written in such a way that the answer choices are balanced, and eliminating one answer choice 
would cue students toward either a correct or incorrect answer choice. A simple example is a 
problem that asks the student to multiply 2 by —2, with the following response options: (a) -4, 
(b) —2, (c) 2, (d) 4. In this case, there is no way to remove a response option without leaving an 
answer that shares one trait (either the sign or the value) with each of the other two, a cue that is 
likely to attract a test taker. Such a modification would likely change the properties of an item. 

One practice that is allowable but has not been used yet by states is the option to retest students 
and use the best of multiple scores. Such a practice would not change the construct being reflected 
by the scores, except to the degree that a student might improve from practice, or that the two 
forms of a test might not be equivalent. On an individual basis, retesting and using the higher 
score would result in an equal or higher score being reported, and on a group basis the mean 
score would be higher. The drawbacks to this practice include a relatively high cost, both fiscally 
and with regard to time, and that it does not address the goal of developing tests that are more 
appropriate for the eligible population. Although retesting and using the average would yield 
a more reliable set of scores, based on calculation from a larger number of items, using the 
higher score effectively eliminates the contribution of all of the items on the test that is not used. 
Measurement would only be improved in this case if students systematically received higher 
scores on a test that was a superior measure. 


Lessons Learned and Questions Generated 


In a relatively short time doing research inspired by the AA-MAS, we have learned a number 
of key lessons that have in turn evoked a number of new questions. One of the first lessons 
that we learned was that the process by which these AA-MAS are to be developed is, as yet, 
unnamed. Although we prefer modification as a conservative term, others prefer the more positive 
enhancement, or the more general alteration. Through research, we have learned that the criteria 
for eligible students in the federal regulations will identify a population of students who achieves 
at-a lower level (Elliott et al., 2009) and reads at a slower rate than do SWODs or SWD-NEs. 
We have also learned that a theoretical and data-based approach to modification can yield items 
and tests that are as reliable as the originals (Kettler, Rodriguez, et al., 2009). These modified 
items and tests are easier for all three groups (Elliott et al., 2009), and the effect is differential in 
favor of the eligible group, when using a design that controls for differences in ability. Also, we 
have learned that reducing the number of words in an item stem may be a helpful modification, 
while caution should be used before adding pictures to reading tests (Kettler, Rodriguez, et al., 
2009). Last, through research projects and interactions with colleagues on the topic of designing 
AA-MAS, we have witnessed the development of a modification paradigm. Although unlikely to 
solve all of the issues raised by the AA-MAS policy, this evolving paradigm promises to advance 
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the conversation by providing a common framework and language for testing professionals who 
are considering these issues. 

Questions that remain unanswered include (a) how to define and maintain a grade-level test 
during modification, (b) whether grade-levelness applies solely to content standards or extends 
also to item difficulty, (c) whether the modified tests reflect the same theoretical structure that is 
reflected by the tests in their original conditions, (d) whether reading load should be reduced on a 
reading test and how to do so, (e) what are the long term consequences of modified examinations 
and the AA-MAS policy, and of course (f) what will we name the process used to create an 
AA-MASs? These questions reflect a commitment to evaluating AA-MAS with the same types of 
evidence used to evaluate any educational measurement tools, and pursuing their answers through 
a theoretical and data-based modification paradigm, as our best strategy for designing tests that 
better measure eligible students’ achievement and yield access to grade-level content. 
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Alternate assessments of modified academic achievement standards (AA-MAS) must be designed, 
developed, implemented, and validated following the same rigorous principles and procedures used 
for other assessments. However, the uniqueness and unfamiliarity of the target population for these 
assessments requires innovative thinking, especially in regard to writing relevant performance level 
descriptors (PLDs) and setting appropriate achievement standards. In this article we discuss consid- 
erations for (a) likely designs of AA-MAS; (b) the writing of PLDs that are relevant to the target 
examinee population for these assessments; and (c) standard setting methods that would be appropri- 
ate for these assessment designs, as well as special considerations relevant to those standard setting 
methods. 


Alternate assessments of modified academic achievement standards (AA-MAS) require both 
conventional and innovative education achievement testing practices. On one hand, designing, 
developing, implementing, and validating AA-MAS should follow the same technical procedures 
and meet the same technical standards as other grade-level assessments. This is both a com- 
monsense requirement and a stipulation in the Standards and Assessment Peer Review Guidance 
(U.S. Department of Education [USDE], 2007b). On the other hand, achievement test designers 
and psychometricians have little familiarity with the intended target examinee population. Their 
limited experience with students with mild and moderate disabilities—the students we think are 
most likely to be appropriately eligible for AA-MAS—comes from deliberations about providing 
test administration accommodations on grade-level assessments. Test designers and psychome- 
tricians have had little direct and focused experience with who these students are, what they are 
studying, how they learn, and the disabilities that create their persistent learning difficulties. Fed- 
eral guidelines do not define the target population clearly, although they stipulate that AA-MAS 
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extend only to students with disabilities. We do know that as a group these students have a history 
of poor performance on large-scale assessments. 

State assessment program practice in defining the target examinees and developing AA-MAS 
suggests that we have a long way to go. For example, in March 2008 six states submitted evidence 
on their AA-MAS for Title I peer review. No state met all requirements. Further, four general 
shortcomings appeared across all of the state submissions (Filbin, 2008). Two of them are directly 
relevant to this article: 


— 


. Identification of the target examinees for AA-MAS. 
2. Clarification of the relationships among AA-MAS, alternate assessments of alternate achieve- 
ment standards, and the corresponding grade-level assessments. 


In this article we describe five steps for designing, developing, implementing, and validating 
AA-MAS. We describe these steps to reinforce our point that standard procedures for designing, 
developing, implementing, and validating all educational achievement tests apply equally rigor- 
ously to AA-MAS. However, these assessments are unique, as are the students who are intended 
to be eligible for them. Both are somewhat unfamiliar to our field. For these reasons, we focus 
on considerations for (a) possible designs for these assessments; (b) the writing of performance 
level descriptors (PLDs) that are relevant to the target examinee population and content stan- 
dards for these assessments; and (c) standard setting methods that would be appropriate for these 
assessment designs. 


STEPS FOR DESIGNING, DEVELOPING, IMPLEMENTING, AND VALIDATING 
AA-MAS 


The five steps we describe are consistent with conventional educational achievement test practice 
with one exception: defining the target examinee. Usually, defining the target examinee population 
is as simple as defining the grade levels for which achievement tests are intended; however, that 
is not currently the case with AA-MAS. The five steps are briefly listed and then discussed in 
separate sections of this article. 


1. Define the target examinee population and the academic content standards they are pursuing. 
This actually requires three steps: (a) defining the target examinee population, (b) describing 
the grade-level academic content standards that the target examinees are pursuing, and (c) 
describing the grade-level academic content standards that the target examinees should be 
pursuing. We address these three steps together because defining the population involves, in 
part, examining the focus of students’ instruction and the ways that instruction is adjusted for 
students’ individual needs. 

2. Design, develop, and implement the test. Whether the design approach is to modify items from 
an existing grade-level assessment, develop new items from grade-level test blueprints, or use 
other design approaches (e.g., assessment portfolios), items and test forms must be closely 
aligned to grade-level content standards and grade-level test blueprints, provide test admin- 
istration environments that are guided by policies for granting appropriate accommodations, 
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and enable valid inferences about what students know and can do in relation to grade-level 
content standards. 

3. Develop appropriate PLDs. These descriptors should be specific to the target examinee pop- 
ulation. They define levels of performance, including the all-important Proficient level. They 
serve as a policy statement of a state’s goals for student performance in relation to standards 
and represent aspirations for the academic achievement of a segment of the school children of 
a State. 

4. Set performance standards. We discuss different standard setting methodologies and the 
application of these methodologies by the type of test being administered. Some standard 
setting methods are more appropriate for multiple-choice tests, some for tests comprised of 
both multiple-choice and constructed-response items, and some for portfolio assessments. For 
each method, we address challenges to adapting them for AA-MAS. 

5. Conduct validation studies. It is important to collect validation data throughout the test 
development process. We discuss ideas for collecting data external to AA-MAS for validation 
purposes. 


THE TARGET EXAMINEE POPULATION 


The most recent version of the Standards and Assessment Peer Review Guidance (USDE, 2007b) 
provides guidance on defining this unique student population. The guidance suggests that these 
are students with disabilities who may achieve “significant growth” in grade-level academic 
content but who, despite “appropriate instruction ... will not achieve grade-level proficiency 
within the [school year]” (p. 3). For these students, neither grade-level achievement standards 
and assessments nor assessments based on alternate achievement standards are appropriate. 
Although these students pursue statewide content standards at their grade level, they are not 
expected to master the volume, scope, breadth, and perhaps depth of content and skills that 
their grade-level peers are expected to master during a school year. They have been referred 
to colloquially as “gap” students because expectations for their academic achievement is below 
that of their peers in the general population and above that of students with significant cognitive 
disabilities who participate in alternate assessments based on alternate achievement standards 
(e.g., Ferrara, Swaffield, & Mueller, 2009). 

Defining this population of students is an evolving inferential, conceptual, and empirical 
enterprise. Theoretically, many or most of the students who should be eligible for AA-MAS 
would have been diagnosed with moderate-or severe learning disabilities or even mild men- 
tal retardation, and this diagnosis would be cited in their Individualized Education Programs 
(IEPs). Further complicating matters, it is well known that standardization and consistency in 
the identification of learning disabilities is questionable; there may well be students without 
disabilities for whom an assessment of modified academic achievement standards would be most 
appropriate. Also, some students who begin their school careers participating in assessments of 
alternate achievement standards may, through effective instruction, develop academic knowl- 
edge and skills sufficient for them to “graduate” to AA-MAS. From a conceptual point of view, 
students who may be eligible for AA-MAS function academically and achieve mastery of grade- 
level content and skills at levels above students who are appropriately eligible for assessments 
of alternate achievement standards and below students who are appropriately eligible for other 
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grade-level assessments. Thus, one approach to defining this population may be to characterize 
the highest levels of achievement of students with significant cognitive disabilities and the low- 
est levels of achievement of disabled students (and perhaps nondisabled students in the general 
population). 

Some researchers have begun to define the population using empirical approaches. For ex- 
ample, Georgia defined “persistently low performing” students (Fincher, 2008) as those students 
who achieved the lowest performance level on the statewide reading or mathematics grade-level 
assessments for three consecutive annual administrations. This definition identified 3 to 4% of 
fifth graders in one or the other content area, 4% of eighth graders in reading, and 9% of eighth 
graders in mathematics. A subsequent teacher group, after reviewing modified items that had been 
pilot tested for potential use in an AA-MAS and other relevant information, identified several 
characteristics of persistently low-performing students. They suggested that students in this group 
tend to be passive learners, display metacognitive deficits (e.g., they do not generalize skills and 
concepts to new situations and cannot change topics easily), and display limited vocabulary and 
prior knowledge. Although this information is valuable for defining IEP goals, providing instruc- 
tion, and defining target examinees, it provides only limited guidance for identifying item formats 
that may be problematic for target examinees, designing AA-MAS that are aligned with grade- 
level content standards, writing performance level descriptors, and setting modified achievement 
standards. 

In another example of empirical work, a consortium of the Minnesota, Ohio, and Oregon 
assessment programs and the American Institutes for Research has taken a similar approach 
to identifying persistently low performing students. They defined persistently low-performing 
students as in the Georgia study. They then evaluated items with acceptable and unaccept- 
able psychometric characteristics for persistently low-performing students and conducted fo- 
cus groups with teachers to identify learning and performance characteristics of students who 
should be eligible for AA-MAS. In addition, they have written standards, instruction, achieve- 
ment, and assessment (SIAA) profiles (Ferrara & Wright, 2007) of students to define the range 
of achievement and instructional focus for students who they intend to be eligible for AA- 
MAS. These SIAA profiles include the instructional goals, learning activities, and academic 
achievement of the intended range of eligible students. Each SIAA profile addresses social and 
interpersonal skills; instructional goals, academic support, and academic achievement; and ap- 
propriate participation in assessments, accommodations, and modifications. The general strategy 
defines and describes the learning goals and achievement of students at the top and bottom 
of the range of students who should be eligible for AA-MAS, and the range of students in 
between. 

‘From a practical or teaching practice point of view, the content of students’ instructional 
programs may be the best indicator of the need to participate in AA-MAS. Students for whom 
AA-MAS are intended do not simply perform poorly in school and on grade-level assessments. 
AA-MAS target students with disabilities who may or may not be striving in school—but who 
are hindered by persistent learning difficulties when they do strive. Modifications to learning 
expectations and test items are intended to provide these students access and opportunity to 
achieve grade-level academic content. At least for those students whose instruction is individually 
and appropriately tailored through special education services, the focus in instruction should be 
on grade-level academic content standards where expectations for mastering that material is 
adjusted as needed and on participation in AA-MAS. 
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What Gets Modified? 


Speculations and confusion about this question abounded (e.g., Filbin, 2008, p. 8) until peer 
review guidance was updated to address AA-MAS in December 2007 (see USDE, 2007b). The 
answer is quite clear: The academic content for students in this population is defined by grade- 
level content standards. Those grade-level content standards cannot be modified. The updated 
guidelines are quite explicit about this point: “Challenging academic standards . . . [must] be the 
same academic standards that the State applies to all public schools and public school students 
in the State [and] include the same knowledge, skills, and levels of achievement expected of all 
students” (USDE, 2007b, p. 2). 

Expectations for how much of the grade-level knowledge, skills, and understandings students 
are likely to master in a school year can be modified because it is appropriate and realistic to 
do so. It follows logically that the achievement standards established by the cut scores on these 
assessments and the “descriptions of the competencies associated with [each] achievement level” 
(USDE, 2007b, p. 3) can and should be different from the grade-level achievement standards for 
the grade-level assessments. 

It is interesting to think about defining achievement standards for an entire subgroup of special 
education students. The bedrock philosophy and principle for delivery of services for students 
with disabilities in special education is individualization. However, the development of modified 
achievement standards and assessments for a group of students with disabilities does not have to 
be viewed as a philosophical shift. Rather, it is public illumination of what academic content this 
distinguishable group of students should be taught and should be learning, and how much of that 
content knowledge, skills, and understandings they should be expected to learn each school year. 

Likewise, it follows that the content and design of AA-MAS can and should be modified. We 
address this topic next. 


POSSIBLE DESIGNS FOR AA-MAS 


Designing an assessment system for the target examinees for AA-MAS continues to be a challenge 
for states. The Federal nonregulatory guidance (USDE, 2007a) provides approaches that states 
might consider to meet the needs of students in this group. The nonregulatory guidance requires 
states to assure that the assessments are built following commonly accepted test development 
processes, provide accessible tests, and ensure reliable and interpretable results. The guidance 
also states that the content standards used for AA-MAS are to be the same grade-level content 
standards taught and tested in grade-level assessments. 

States can operationalize this guidance in different ways when they define achievement stan- 
dards and test designs. States may choose to create modified academic achievement standards 
that are disconnected from grade-level achievement standards or may link to them conceptually 
via a standard setting process. The guidance requires states to assure that modified academic 
achievement standards are appropriate and meaningful for target examinees. States must include 
at least three performance levels (e.g., Basic, Proficient, and Advanced) and the number of levels 
does not have to be the same number of levels as for the corresponding grade-level assessment. In 
addition, states cannot simply set lower cut scores on grade-level assessments. The guidance does 
not stipulate test designs. States may include multiple-choice items, constructed-response items, 
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portfolio designs, and other approaches. States may also choose to develop AA-MAS by modi- 
fying grade-level assessments or by developing entirely new assessments. States are required to 
ensure that blueprints for AA-MAS are comparable to blueprints for the grade-level assessments 
(Rigney, 2008; USDE, 2007b, pp. 25—26) 


Current Practices 


It appears that, so far, states are not choosing to create completely new assessments (Lazarus, 
Thurlow, Christensen, & Cormier, 2007). Some states are modifying existing grade-level test 
item pools or intact test forms. In line with the examples provided in the nonregulatory guidance, 
current approaches to modifying grade-level assessments include eliminating an incorrect answer 
choice in multiple-choice items, providing shorter reading passages or segmenting them, using 
plain English principles to simplify items, and/or requiring fewer test items. 

Given the current economic climate and the erosion of the tax base in many states, the 
modification of a state’s current grade-level assessment may prove a popular option as states seek 
affordable ways to develop AA-MAS. For these states, AA-MAS with item types that may be 
machine scanned (e.g., multiple-choice items) as opposed to open-ended items and portfolios 
that require human scorers may also be preferred. Although some states may simply opt out of 
creating AA-MAS, the political realities may force other states to develop AA-MAS. 

A report by the National Center on Educational Outcomes (Lazarus et al., 2007) about the 
six states that reported in 2007 that they had developed or were developing AA-MAS, found the 
following: 


© One state used a portfolio approach. 
e The other five states used a traditional testing approach and modified their grade-level 
assessments: 


e All five states used multiple-choice items, two also included writing prompts, and one 
included constructed-response items. 

e Some states included items only at the lowest two depths of knowledge levels; most 
states removed an incorrect answer choice, used fewer test items and simplified test 
language, and/or used fewer and shorter reading passages (Lazarus et al., 2007). 


Other states have focused on improving the accessibility of their grade-level assessments 
(e.g., using plain English criteria) rather than creating new tests or modifying existing items. 
For example, Colorado has done extensive work to increase the accessibility and alignment 
with universal design principles of their existing grade-level assessments and ensuring that their 
accommodation guidelines and implementation of accommodations in the field are maximized 
and appropriate (see HB 05-1246 Study Committee, 2005). This falls directly in line with the 
nonregulatory guidance (USDE, 2007a). Other modification approaches are not consistent with 
the nonregulatory guidance. For example, selecting reading passages that are not on grade level 
is considered testing off grade and not acceptable. We note that the determination of what is 
“srade level” is an oft varied and complex task. Minimizing the breadth of standards covered on 
an AA-MAS is considered misalignment and not acceptable. 
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Technical Quality 


Whether states choose to retool existing assessments (a seemingly economical solution) or create 
new assessments, AA-MAS must meet conventional standards for rigor and technical quality. 
They must be aligned to grade-level content standards, the accompanying modified achievement 
standards must be well articulated, and both must enable reliable and valid inferences about the 
academic achievement of grade-level content standards for eligible students. 

Current practices in test alignment and standard setting provide guidance on the length of 
AA-MAS. For example, one might consider no fewer than six items per content standard (Webb, 
2006) and as few as 10 items or as many as 20 per achievement level. Test score reliability is 
a consideration for test length as well. States should consider the amount of testing time that 
students will need to respond to a full test form, provide sufficient breaks particularly if the test 
is long, and consider the impact on score reliability of shortening or lengthening tests. 

States also will need to consider whether item designs are appropriately accessible for target 
examinees. For example, multiple-choice items that require multiple steps to achieve a solution 
may be more appropriately presented as multipart items on AA-MAS (unless of course the item 
is intended to measure multistep problem solving). These considerations may require different 
proportions of item types on AA-MAS compared to the proportions on the regular grade-level 
assessments, and rightly so. As we begin to understand the population, we may find that a 
distinguishing factor between students taking AA-MAS and students taking regular grade-level 
assessments is that AA-MAS students possess content knowledge but are less proficient at 
problem solving, thinking critically, and reasoning than students taking the regular grade-level 
assessments, as suggested by the Georgia study (Fincher, 2008). 

Until new AA-MAS are actually administered to the target examinees, it is difficult to make 
assumptions about how items and test forms will perform psychometrically. For example, in the 
design approach involving elimination of a multiple-choice distractor, it is important to review 
data on the grade-level assessment based on students presumed to fall into the target group and 
find which distractors are most troubling or least useful for that group, as opposed to looking at 
distractor data only for the full examinee population (Barton, 2006). States should ensure that 
the final version of the assessment follows standard steps and procedures for test development, 
adapted for AA-MAS, as discussed previously. 


WRITING PLDS FOR AA-MAS 


The PLDs! articulate the academic knowledge and skills that students in a particular achieve- 
ment level are expected to be able to demonstrate (Lewis & Green, 1997; Perie, 2008). As such, 
PLDs inform students, their families, educators, and the general public about the types of knowl- 
edge, skills, and abilities expected of, say, Proficient students in a particular content area. This 
straightforward purpose of PLDs obscures the controversy surrounding their development. Some 
researchers assert that PLDs must be developed at the beginning of the test-development cycle 
(Loomis, 2001; Perie, 2008), whereas others argue that PLDs should only be written following the 


"Also known as achievement level descriptions in the standard setting literature and peer review guidance. We use the 
terms achievement level and performance level interchangeably. 
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completion of the standard setting (Lewis & Green, 1997). Peer Review guidance for NCLB has 
settled this point in K-12 testing, requiring that the initial PLDs are developed prior to standard 
setting. We contend that both processes for developing PLDs have merit and can be used to the 
benefit of the standard setting and the testing program (Mercado & Egan, 2005). 

By developing PLDs at the front end of the test development cycle, items can be written that 
target and align with the descriptors (e.g., Bejar, Braun, & Tannenbaum, 2007). These initial 
PLDs should be developed to provide the state’s expectations for examinee performance. For 
standard setting, the initial PLDs can be used to guide standard setting panelists. The cut scores 
from a standard setting are an operationalization of the state’s PLDs. By refining and validating 
those initial PLDs after the cut scores have been established, we can be certain that the final PLDs 
reflect actual student knowledge and performance. 

In the next section, we discuss the development of the PLDs for AA-MAS; specifically, we 
discuss four aspects that states should consider when developing PLDs in the context of AA- 
MAS: (a) the relationship to the grade-level assessment, (b) policy-based PLDs, (c) test-based 
PLDs, and (d) final PLDs. 


Relationship Between PLDs for AA-MAS and Grade-Level Assessments 


In developing the PLDs for AA-MAS, the first step must be to consider the relationship between 
the PLDs for grade-level assessments and the ones to be developed for AA-MAS. The USDE 
(2007a) nonregulatory guidance allows for a different definition of Proficient on the AA-MAS 
even though the same grade-level content standards must be used for both the grade-level as- 
sessments and AA-MAS. According to the Standards and Assessment Peer Review Guidance 
(2007b), “modified academic achievement standards must—be challenging for eligible students, 
but may be less difficult than the grade-level academic achievement standards” (p. 17). Given that 
AA-MAS are meant to be a less difficult version of a state’s grade-level assessments, it is only 
logical that the AA-MAS proficiency definition will reflect that decrease in difficulty. In other 
words, students taking the AA-MAS will most likely be expected to master less grade-level con- 
tent than students taking the regular grade-level assessments. As states move forward with both 
assessment types, they will want to consider how to communicate the differences in expectations 
for Proficient on AA-MAS and grade-level assessments. 

In part, these differences in expectation may be communicated by using different performance 
level labels and even a different number of levels. The nonregulatory guidance clearly states 
that states need not use the same number of performance levels for AA-MAS as for their grade- 
level assessments (USDE, 2007a, p. 23). As mentioned earlier, this guidance requires at least 
three performance levels. Elliott, Kettler, and Roach (2008) argued that changing the number 
of performance levels between grade-level assessments and AA-MAS may prove confusing for 
stakeholders, and they questioned how the state would decide which performance level to drop. 

This is a fair question. If a state currently uses four performance levels (e.g., Below Basic, 
Basic, Proficient, and Advanced), the highest performance level (e.g., Advanced) seems likely to 
be dropped. If this were to happen, it is easy to imagine that stakeholders may become upset that 
eligible students do not have the opportunity to achieve the Advanced level on AA-MAS. On the 
other hand, it may prove more beneficial to the end users of AA-MAS if the underlying achieve- 
ment spectrum is partitioned differently than the achievement spectrum underlying grade-level 
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assessments. It is anticipated that students intended to be eligible for AA-MAS are able to master 
grade-level content but at a slower pace than other students (Elliott et al., 2008). Thus, we might 
expect that fewer students will attain the Proficient level on AA-MAS. If this is the case, it may 
make sense to partition the Basic achievement level into two categories to provide opportunity to 
students who have not yet achieved the Proficient level to show improvement. 

There are also psychometric considerations regarding the number of performance levels. 
Shorter tests, like AA-MAS, provide less information about student performance, making it de- 
sirable to segment the student achievement continuum into fewer achievement levels; moreover, in 
planning for item-based standard setting methods (e.g., Bookmark, Item Descriptor [ID] Match- 
ing), sufficient numbers of items should be present along the continuum of student achievement to 
enable accurate description of content expectations within each achievement level and to support 
the breadth and depth of descriptors for each achievement level. 

Ercikan and Julian (2002) provided guidance on the number of performance levels by showing 
the degree of classification accuracy that state assessment programs can expect to achieve on an 
assessment based on the number of achievement levels and test score reliability. As they pointed 
out, the greater the number of achievement levels planned for a test score scale, the lower the 
accuracy will be of correctly identifying the true achievement level of examinees. The same 
relationship also exists between score reliability and classification accuracy. 

Should the same or different performance level names be used for AA-MAS as for grade- 
level assessments? Beyond saying that three performance levels must be established, the USDE’s 
(2007a, 2007b) guidance does not specify that the same labels should be used for AA-MAS 
performance levels as for grade-level assessments. Even though AA-MAS assess the same grade- 
level content, they assess less rigorous content. If a state uses the same labels for the performance 
levels, stakeholders could confuse the two sets of performance descriptors. The category Proficient 
will mean something entirely different on AA-MAS than it does on grade-level assessments. It 
seems appropriate to use different performance level names for AA-MAS to decrease possible 
confusion between the two tests. 


A Process for Developing PLDs 


There will probably never be one best process for developing and validating PLDs, and states 
have taken a variety of approaches to conceptualizing and defining Proficient and other levels of 
performance (see, e.g., Ferrara et al., 2009). In this section, we describe a three-step process for 
developing and refining PLDs. Throughout this process, the intent is to clarify, refine, and validate 
the PLDs. The intended meaning and interpretation of PLDs must remain intact throughout the 
process of setting cut scores and refining the PLDs. 


Policy-Based PLDs Prior to Standard Setting.? After determining the appropriate num- 
ber of and names for achievement levels, the state’s next step is to conceptualize the level of 


2We refer to policy-based PLDs that are based on and refer explicitly to grade-level content standards. These 
policy-based PLDs are distinct from policy definitions that apply generically to all grades and content areas, like 
those developed for the National Assessment of Educational Progress. (See http://www.nagb.org/policies/PoliciesPDFs 
/Technical%20Methodology/developing-student-performance.pdf) 
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proficiency they expect from students in each achievement level. These policy-based PLDs can 
be developed directly from the content standards, and outline the expectations that the state holds 
for what students should know and be able to do based upon the grade-level content standards. 
In recent years, and because of the influence of the peer review process, policy descriptors have 
evolved from generic performance statements (e.g., National Assessment of Educational Progress 
policy definitions) to content-based descriptions. 

These more recent PLDs typically reflect the entire range of performance within an achieve- 
ment level and do not target a particular area of the achievement level such as borderline perfor- 
mance. They assert academic knowledge and skills that the state expects of the Proficient student 
and students at other performance levels. They also set the tone for the standard setting (as well 
as the tone of the testing program). For AA-MAS, the tone of the policy-based PLDs should 
reflect the tone of the PLDs for the grade-level assessment. For example, one assessment should 
not assert world class standards, whereas the other calls for standards that represent minimal 
competency. 


Developing Policy-Based PLDs. When developing PLDs for AA-MAS prior to setting 
standards, there are three areas that the state should consider: (a) the grade-level assessments’ 
PLDs, (b) the content standards, and (c) the articulation of PLDs across grade levels. Further, 
development of any PLDs requires the input of knowledgeable stakeholders, such as assess- 
ment specialists, content specialists, and specialists in teaching students with disabilities. The 
development of PLDs may be accomplished through formal workshops or through informal 
meetings between the state and stakeholders. The remainder of the discussion in this section 
assumes a formal workshop is held; however, these steps could easily be applied to an informal 
meeting. 

The grade-level assessment PLDs should serve as a starting point for the development of 
policy-based PLDs for AA-MAS. The content knowledge and skills targeted in the grade-level 
PLDs can be discussed in relationship to the students taking AA-MAS. During this exercise, 
workshop panelists can discuss the relationship between the achievement levels for grade-level 
assessments and AA-MAS. Panelists should pay explicit attention to the rigor as well as the 
content expectations asserted in the PLDs for grade-level assessment so that they are able to 
explicate if and how the rigor and content expectations have changed (or are the same) in the 
PLDs for AA-MAS. 

Deconstruction of the content standards is another important component to the development 
of policy-based PLDs. During this process, workshop panelists consider each content standard 
and how students in each achievement level may perform on the content standard. The content 
standard may be parsed as panelists discuss how students at different performance levels are 
expected to perform on portions of the standard. For example, as panels deconstruct the range of 
inferences that can be made from informational and literary texts and how various texts influence 
the complexity of the inference, they can begin to identify inference types they expect to be 
present in particular performance levels on AA-MAS. Once the content standards have been 
deconstructed, panelists should examine the articulation of expected academic knowledge and 
skills across the achievement levels. Panelists should check that the academic knowledge and 
skills are coherent across the performance levels, and that the level of expectations increases as 
the achievement levels increase. 
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As a final step in the development of policy-based PLDs, a meta-panel should examine the 
PLDs for the articulation of expected academic knowledge and skills across the grade levels. 
The meta-panel would consist of representatives from the grade-level panels. This panel should 
ensure that the expectations of the academic knowledge and skills increase meaningfully from 
grade to grade. 


Test-Based PLDs. The policy-based PLDs summarize the state’s expectations of student 
performance in each achievement level. They articulate what students should do to be in a partic- 
ular achievement level. The test-based PLDs, on the other hand, summarize student performance 
and how students did perform in each achievement level. Ideally the two would be one and the 
same; however, research has shown that policy-based PLDs are not always an accurate reflection 
of student performance once tests are administered (Burstein et al., 1996; Mercado & Egan, 2005). 
By themselves, policy-based PLDs may provide a somewhat inaccurate picture of what students 
in each achievement level know and can do and may confuse parents and teachers regarding the 
content knowledge and skills students actually hold. 

Although policy-based PLDs reflect the range of student performance within an achievement 
level, test-based PLDs focus on a specific part of the range, such as the threshold or middle of the 
achievement level. Test-based PLDs should reflect the tone of the policy-based PLDs. Because 
test-based PLDs describe the knowledge, skills, and abilities found in the test items, they are 
sometimes considered a snapshot of the skills held by students in each achievement level. 


Developing Test-Based PLDs. The development of test-based PLDs begins with standard 
setting, when panelists deconstruct the policy-based PLDs to focus on a target student. The target 
student is the student for whom cut scores are being set and provides a starting point for standard 
setting panelists. The discussion of the target student for each performance level helps panelists 
come to a common understanding of the expectations that they will operationalize through the 
standard setting. When panelists first discuss target students, they use three sources of information: 
their own knowledge, the content standards, and the state’s policy-based PLDs. In the case of 
AA-MAS, it may also be helpful to provide panelists with summaries of the differences between 
the expectations for students on the grade-level assessments and the AA-MAS as well as the 
differences in the student population taking the two tests. 

Throughout the standard setting, panelists gain greater insight about the target student through 
discussion with their peers, analysis of the items, and exposure to impact data, which may 
influence their original description of the target student. As such, the target student is a dynamic 
product that panelists discuss throughout the standard setting. At the end of the standard setting, it 
is important for panelists to have time to finalize the target student description based on everything 
gleaned during the standard setting. 

The format of the final session will depend on the standard setting procedure used. This is a 
routine step in the Bookmark Standard Setting Procedure: panelists write descriptors of the items 
that precede the cut score. As an aggregate, these items describe the student who just enters the 
achievement level, that is, the skills of the target student. With other standard setting methods, it 
may be necessary to have the items mapped so that panelists can describe them. 

States that use an item response theory (IRT) model to scale their assessments can collect 
information regarding how the knowledge, skills, and abilities targeted by items that map to 
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TABLE 1 
Partial Item Map With Completed Knowledge and Skill Descriptors 
Order of Difficulty Item No. Knowledge and Skill Descriptors 
AL3 38 75 Draw conclusions from informational text 
3 35 Draw conclusions from informational text 
36 67 Identify supporting detail in informational text (persuasive letter) 
35 78 Draw conclusions from informational text 
34 62 Draw conclusions from informational text 
33 15 Draw conclusions from informational text 
32 89 Locate specific information in informational text (table) 
31 vp Draw conclusions from narrative with graphics 
AL2 30 86 Identify main idea in informational text 
29 84 Locate specific information in informational text 
28 64 Determine the author’s purpose 
oy 76 Identify supporting detail in informational text. 
26 DZ Identify supporting detail in informational text (narrative) 
25 31 Identify main idea in informational text (narrative text) 





an achievement level score range. Mapping the items this way relates item demands to the 
descriptions of what students should know and be able to do, regardless of the standard setting 
method used. After the operational standard setting is completed, an analysis for each item may be 
conducted in which a short phrase termed an item descriptor is developed. These item descriptors 
can be compiled into a PLD that reflects the target student. Because AA-MAS may be relatively 
short, supplementing final PLDs with illustrative items could be particularly helpful. Table 1 
shows an example of an item map that has been partially completed with item knowledge and 
skill descriptors. 


Final PLDs. The final PLDs should reflect both the policy- and test-based PLDs. Both 
types of PLDs provide different types of information regarding student performance in each 
achievement level. In this final phase, the two types of PLDs can be combined so that the final 
PLDs reflect what students in each achievement level are expected to (should) know as well 
as examples of actual academic knowledge and skills currently held by the students (i.e., what 
students can do). During the final phase, it may be necessary to update the policy-based PLDs to 
reflect knowledge gained from the development of the test-based PLDs. 


Developing Final PLDs. The development of the final PLDs provides the state with the 
opportunity to synthesize the ideas and information found in the policy- and test-based PLDs. 
This process may reveal areas of disconnect between the two types of PLDs where academic 
knowledge and skills that students should have (via the policy-based PLDs) are not supported by 
the test-based PLDs. To finalize the PLDs, the state will want to have stakeholder input either 
through formal or informal meetings. 

The first step in finalizing PLDs is to compare the skills in the two PLDs, side by side, for 
concurrence. If a skill was expected at the Proficient level in the policy-based PLDs but was 
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Guidance for the interpretation of these descriptors: The Performance Level 
Descriptors in this document give details about the knowledge and skills that students at 
the cut score for that particular achievement level are expected to know and be able to do. 
The particular standards and items that are tested may change from year to year to ensure 
comprehensive coverage of the state’s standards as a whole. The skills described below 
indicate summary statements of skills that we expect students to consistently 
demonstrate, but these should not be interpreted as the only skills measured on these 


assessments or the only knowledge and skills students at the cut score possess. 
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FIGURE 1. Sample interpretive guidance. 


easily mastered by Basic students according to the test-based PLDs, then this conflict will need 
to be examined. It may be that the skill simply needs to be re-aligned to the Basic category in the 
policy-based PLDs, or it may be that the skill needs to be further deconstructed because it was 
assessed only by items that captured a simpler component or depths of knowledge level of the 
skill. 

Field testing PLDs may be an appropriate final step prior to release of the final version of 
the PLDs. First, if a state sets standards on one test form, a field test period would allow the 
state to validate the PLDs with data from a second test form to provide more stable information. 
Second, panels of teachers could review the PLDs and their interpretive guidance for clarity 
and understandability. Figure 1 is an example of suggested interpretive guidance. Teachers could 
provide feedback regarding how helpful (or not helpful) the PLDs are in providing useful feedback 
about student abilities. 


PLDs Over Time. States must be very clear in how they define final PLDs, especially 
the skills from the test-based PLDs. The skills described in a test-based PLD often reflect a 
single form of the test. Such PLDs are a single snapshot in time of the skills that students in 
each achievement level can do. If a state does not plan to revisit the PLDs, then it is important 
that stakeholders are informed that the PLDs reflect a subset of skills held by students in each 
achievement level. In this case, PLDs are seen as static definitions that do not change as new 
information becomes available. 

In other cases, PLDs are seen as fluid definitions that can and should be updated as new test 
forms are administered to students. Each new form will provide more information regarding the 
skills that students can do. Schneider, Egan, Kim, and Brandstrom (2008) found that only 19% to 
30% of the original PLDs remained stable over time as new items were introduced in subsequent 
test forms. This suggests that original PLDs are limited in their ability to reflect actual student 
performance. This research echoes advice from Crane and Winter (2006), who recommend that 
PLDs be reviewed periodically to ensure that the descriptors still align with the competencies 
of the test. We also recommend including a discussion during test-based performance descriptor 
writing that reflects not only the test form(s) included in standard setting but also the content 
standards and items that are expected to appear in subsequent forms. 

All of this points to the need for policy-based and test-based PLDs to work hand in hand. 
Policy-based PLDs are often more extensive than test-based PLDs and may be able to withstand 
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the differences in test forms over time. It is important that there is alignment between the policy- 
based and test-based PLDs. 


SETTING MEANINGFUL AND APPROPRIATE MODIFIED ACHIEVEMENT 
STANDARDS 


Setting cut scores and establishing modified achievement standards is the culmination of the 
design and implementation process of a new and unique assessment program and is the final 
step prior to reporting of assessment results. The standard setting design should be based on 
consideration of the test design, the test length, and the way the test is scaled and student 
performance reported. In addition, particularities of AA-MAS deserve special consideration 
when designing a standard setting. These special considerations must be addressed regardless of 
the standard setting method to be implemented. 

In this section, we discuss two special considerations to take into account when planning 
standard setting for AA-MAS. Then we address general considerations that are relevant to all 
standard setting methods. Finally, we examine standard setting methods appropriate for AA- 
MAS comprised of multiple-choice and constructed-response items and methods appropriate for 
portfolio assessment designs. 


Special Considerations for Setting Performance Standards for AA-MAS 


The AA-MAS are based on the same content standards as a state’s grade-level assessments. 
Expectations for what students should know and be able to do, however, are modified for students 
who are eligible for AA-MAS. In turn, the complexity of the PLDs and the difficulty of the test 
items are also adjusted to be consistent with the modified achievement standards. This presents 
a unique situation. The same content standard structure underlies two assessment programs that 
are administered to two different groups of students—supporting the idea that both assessments 
target the same achievement construct. A link exists between the two assessments that should be 
addressed as part of standard setting. 

Standard setting panelists should understand the relationship between AA-MAS and grade- 
level assessments. This can be addressed by the state having a clear definition of the target 
population for AA-MAS and explaining the relationship between the two student populations 
for whom the tests are designed. This also can be addressed through a study of the policy-based 
PLDs, which should be unique for each of the assessments. As a first step for any standard 
setting method, panelists should actively discuss the policy-based PLDs. In the case of AA- 
MAS, the panelists should study the policy-based PLDs for the modified achievement standards 
and compare them to the policy-based and test-based PLDs for the grade-level assessment. In 
most states, the policy-based and/or test-based PLDs for the grade-level assessment have been 
published and disseminated for several years. This comparison activity will aid panelists in 
understanding the difference in expectations between the two assessments. It should be noted that 
this is an additional activity on the meeting agenda that, at least up to now, would not be seen at 
a standard setting for a grade-level assessment. 
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Some states may attempt to link the AA-MAS to the grade-level assessments using statistical 
linking methods so that they can locate the grade-level assessments’ cut scores on the test scale 
for the AA-MAS. For example, a state may use equipercentile equating to establish cut scores 
on the AA-MAS that produce similar impact data for both assessments. Alternatively, a state 
could link the scales from the two assessments in a random sample of students from the general 
examinee population that takes both the AA-MAS and the grade-level assessments. With grade- 
level assessments’ cut scores projected onto the scale of AA-MAS, standard setting panelists 
could review and adjust the cut scores through a review process instead of a full standard setting. 
This approach enables panelists to review and consider the specific attributes of AA-MAS and 
the target examinee population while showing them where the cut scores for the grade-level 
assessments would be located. 

If a state attempts to link AA-MAS and grade-level assessments statistically, it is even more 
important that the standard setting panelists understand the similarities and differences between 
the two assessments. Almost all of the standard setting methods we consider can be used for a 
cut score review as well as for a full standard setting. Of the methods we consider, however, the 
item mapping methods such as the Bookmark Standard Setting Procedure and ID Matching best 
lend themselves to cut score reviews. 


General Considerations for Setting Cut Scores for AA-MAS 


Some aspects of setting performance standards for AA-MAS are universal to all assessments, 
regardless of a test’s format. 


Recruiting Standard Setting Panelists. Two questions regarding panelist recruitment 
must be answered: Who? and How many? The answer to these questions is, A diverse group 
that is as large as possible. For educational achievement assessments, Hambleton and Pitoniak 
(2006) recommended 15 to 30 panelists per standard setting committee to obtain adequate 
representativeness and stability of cut score recommendations. 

The importance of diversity of a standard setting panel has long been recognized (Ham- 
bleton & Pitoniak, 2006). It is also important that all panelists are knowledgeable about the 
students who are assessed and the content standards on which they are assessed (Plake, 2008). 
Obviously, educators who teach the students who are appropriately eligible for AA-MAS are 
most knowledgeable. It is, however, important to recruit panelists from other stakeholder groups 
because a diversity of perspectives enhances the group decision-making process (Surowiecki, 
2004). The group of panelists should represent the views of all stakeholders, including teachers, 
parents, students, administrators, and policymakers. Panelists for the AA-MAS standard setting 
may include teachers of students with learning disabilities, teachers of English-language learn- 
ers, teachers from regular education classrooms, interested members of the public with some 
requisite background knowledge of test content and the target examinees, administrators, and 
parents. The views of education policymakers who are responsible for establishing expectations 
for student achievement—for example, state board members—should be represented as well. 
Elliott et al. (2008) cited the recent Individuals with Disabilities Education Act (commonly 


known as IDEA) that expects a diverse group of stakeholders be involved in the standard setting 
process. 
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It is important to underscore that panelists should have knowledge of the content standards 
and student population for whom the cut scores are being established. As Plake (2008) pointed 
out, the validity of a standard setting may be undermined if the panelists do not have the 
requisite knowledge base prior to entering the standard setting. Although we agree with Plake 
that knowledge of the content and student are important, we believe that the diversity of the panel 
is of equal import. A diverse group of standard setting participants ensures that many different 
points of view will be brought to the table and reflected in the cut scores. For example, parents 
of students taking the AA-MAS may not have content knowledge, but they have advocated for 
that student through IEP meetings, parent-teacher conferences, and so on. They bring a unique 
perspective of the student experience to the table that will differ from teachers. Notice that we 
do not argue that content or student knowledge should be entirely disregarded for the sake of 
diversity; however, we do believe that a more informed decision will be made only by bringing the 
various stakeholder groups to the table. This being said, all panelists must be trained to develop an 
adequate understanding of the content standards, target examinees, and standard setting procedure 
that will be implemented. 

Standard setting panels should be diverse, representing the population of the state. Rationales 
that guide recruitment should be noted explicitly in the standard setting technical report, and 
analyses of panel diversity should be documented. Some states have great difficulty in recruiting 
minority teachers. In these cases, a state may consider basing their recruiting efforts on the 
demographics of the students that educators teach instead of the demographic characteristics of 
the educators themselves. 


Vertical Articulation. The AA-MAS can be administered in multiple grades and content 
areas, making the articulation of the cut scores and/or impact data an important consideration for 
standard setting. Procedures for vertical articulation of cut scores are described elsewhere (e.g., 
Cizek & Bunch, 2007; Ferrara et al., 2007; Hambleton & Pitoniak, 2006; Lewis & Haug, 2005). 
Any standard setting design should consider the issue of vertical articulation and how it will be 
addressed. For AA-MAS, the considerations are unique. For example, many of the earliest states 
have developed AA-MAS for a subset of contiguous grades rather than Grades 3 through 8 and 
high school, and one content area rather than both reading and mathematics. Such states should 
consider articulating achievement standards across the subset of grades when they set standards. 
They will also need to consider how to articulate the achievement standards in the future if 
they add grades and content areas. Current procedures for articulating standards (e.g., writing 
articulated PLDs, articulating impact data by adjusting cut scores; see Ferrara et al., 2007) can 
be adapted. 


Standard Setting for AA-MAS Composed of Multiple-Choice and/or 
Constructed-Response Items 


As mentioned previously, of the six states that have built or are currently building AA-MAS, 
five used only multiple-choice items or multiple-choice items with constructed-response items 
or writing prompts. Assessments with multiple-choice items and those with both multiple-choice 
and constructed-response items lend themselves to conventional standard setting methods (e.g. 
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Bookmark, modified Angoff, contrasting groups). This is not surprising. The predominant ap- 
proach to developing AA-MAS is to adapt items and test blueprints from existing grade-level 
assessments. We provide brief overviews of the methods we cover but leave out important details 
(e.g., when panelists share insights, when they make independent judgments) that are covered 
extensively elsewhere (e.g., Cizek & Bunch, 2007, Hambleton & Pitoniak, 2006). We propose 
approaches to using conventional standard setting methods to meet the unique challenges of 
AA-MAS. 


Bookmark Standard Setting Procedure and Other Item Mapping Methods. The Book- 
mark Standard Setting Procedure (Lewis, Mitzel, & Green, 1996) is one of the most popular meth- 
ods currently used in state assessment programs (Karantonis & Sireci, 2006). Other item mapping 
methods that have been used in other assessment programs include the Item-Mapping method 
(Wang, 2003) and ID Matching (Cizek & Bunch, 2007; Ferrara, Perie, & Johnson, 2008). These 
three methods are related psychometrically because they required panelists to make judgments 
about items mapped to IRT score scales. Their cognitive-judgmental tasks are quite different 
(e.g., Ferrara et al., 2008). Although our discussion about setting cut scores for AA-MAS focuses 
on the Bookmark Procedure, the discussion is applicable to the Item-Mapping and ID Matching 
methods. 

The Bookmark Standard Setting Procedure is typically implemented in three rounds of judg- 
ments using two key components: the Ordered Item Booklet (OIB) and the item map. The OIB 
is composed of multiple-choice and constructed-response items, ordered by item difficulty. The 
item map provides details on the items in the OIB (e.g., item difficulty, scoring key, and the 
content standard that each item targets). Standard setting panelists record their responses or re- 
actions to two questions on the item map or in the OIB: (a) What does this item measure? That 
is, what does a student need to know and be able to do to respond successfully to this item or 
score point? and (b) Why is this item more difficult than the preceding items? In Rounds 1, 2, 
and 3, a panelist places a bookmark on a page in the OIB to indicate a recommendation for a cut 
score. The panelist’s cognitive-judgmental task is stated as “Place your bookmark on the page 
where two-thirds (or one-half) of those students who are just barely in a performance level would 
respond successfully.”? Typically, panelists view impact data (e.g., the percentage of students at or 
above the Proficient level) to inform their final bookmark placements. Final cut scores are based 
on the median of the page numbers on which panelists place their bookmarks in the final round of 
judgments. Full descriptions of materials, training, and procedures for the Bookmark Procedure 
appear in a number of sources (Cizek & Bunch, 2007; Lewis, Green, Mitzel, Baum, & Patz, 1998). 


Challenges in Using the Bookmark Procedure for AA-MAS. Two challenges should be 
considered before implementing the Bookmark Standard Setting Procedure for AA-MAS. These 
challenges must be addressed in a Bookmark standard setting for any assessment; however, they 
require special consideration for AA-MAS. The first challenge is the likely shorter lengths of 


31t is common to state the cognitive-judgmental task in another way: Place the bookmark on the first page where you 
judge that a student who has the knowledge and skills to demonstrate mastery of the items before the bookmark would 
be classified as Proficient [or whatever the achievement level may be]. 
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these assessments; the second has to do with the response probability used to order the items. We 
discuss ways to address these challenges. 


Challenge 1: Test length. In the Bookmark Standard Setting Procedure, the primary focus 
for decision making is the items that comprise the test. Over the course of two or three rounds 
of discussion and decisions, panelists build an understanding of what the target student should 
know and be able to do in each of the performance levels. To do this successfully, panelists must 
view an OIB with enough pages (each multiple-choice item and constructed-response score level 
appears on a separate page) so that they can build adequate understanding of the knowledge and 
skill requirements of the test. In other words, the panelists use this information to build a story of 
what the target student is able to do. The score scale underlying the OIB should not have large 
gaps between item locations. If a test is short, the items may not cover the entire scale range, 
which leaves gaps in the story participants build about what the target student is able to do. 

This is an important consideration for AA-MAS, where states may modify and shorten versions 
of their grade-level assessments. Our experience has shown us that there should be at least 10 and 
as many as 20 items or score points per cut score. This guideline ensures that adequate numbers 
of items appear in the OIJB to cover the contents of the test blueprint, minimize gaps between 
item locations (or difficulty), and provide adequate separation on the test score scale between cut 
scores. Requirements for AA-MAS require at least three achievement levels (i.e., two cut scores). 
This means that OIBs should be between 30 to 60 pages long for AA-MAS. The upper end of 
this range represents a goal, not a requirement. We recognize that it is unlikely that AA-MAS 
will include 60 points. We anticipate that most states will create AA-MAS that are toward the 
low end of our suggested range. 

It is sometimes possible to fill in large gaps in the location of items on the underlying score 
scale by augmenting the OIB with items from the broader item pool. When augmenting, it is 
important to consider covering the entire range of the test scale, the test content blueprints, and the 
gaps in item locations. If a broader item pool is available (perhaps from a field test administration), 
one can begin by identifying items to fill location gaps. At the same time, it is important to avoid 
making the OIB too long and burdensome for panelists to review. This can be done efficiently by 
selecting multiple-choice items or one-point constructed-response items. If the item pool offers 
sufficient choice of items, then it is also important to consider the test content blueprint. To the 
degree possible, the OIB should proportionally match the test blueprint. Often, though, it is the 
case that the available item pool offers a limited number of items and it may be fortunate to find 
any items to enhance the scale coverage. Many assessments do not have item pools from which 
to augment OIJBs. For short tests where the scale is not adequately covered, states may want to 
consider an Angoff or Body of Work standard setting method. 


Challenge 2: The response probability criterion. The response probability (RP) criterion is, 
perhaps, the most discussed aspect of the Bookmark Standard Setting Procedure. In item mapping 
methods such as Bookmark, items are ordered in an OIB using an RP criterion. The selected 
criterion indicates the probability that an examinee whose proficiency on the score scale is equal 
to an item’s location will respond correctly to a multiple-choice item or at a given score level 
on a constructed-response item. The RP criterion also indicates students’ mastery of content 
by specifying the likelihood with which students at a particular scale score will answer items 
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correctly. Using a response probability of 0.67 (RP67) as an example, multiple-choice items and 
score levels for constructed-response items are located on the underlying IRT score scale and 
placed in an OIB so that examinees with proficiencies equal to an item’s RP67 location have a 
two thirds likelihood of answering the item correctly (often, with guessing factored out). Thus, 
when a cut score is placed in an OIB, we can say with some certainty (i.e., two thirds probability) 
that students who meet or exceed the cut score have mastered the content which the standard 
setting participants expected them to master. The students at or above the cut score will have 
at least a two thirds likelihood of answering the item correctly. The RP criterion influences cut 
scores, impact data, and the ordering of the items in the OIB. 

For AA-MAS, the RP criterion should be selected in light of state policy and methodological 
implications. From a policy perspective, what RP criterion is appropriate for students eligible for 
AA-MAS? Should items be ordered using RP67, thus defining content mastery at two thirds? Or 
is a 50-50 (RP5S0) likelihood a more appropriate definition of mastery? Is it important to select 
the same RP criterion that was used for the standard setting for the grade-level assessments? 

The importance of the methodological implications of the RP value cannot be overstated. The 
distributioa and order of items along the test score scale differs depending on the RP value used. 
In some cases, using RP67 locates very few items at the low end of the scale, whereas RP50 
may distribute the items more widely across the scale score range. In this case, RP67 might 
force panelists to choose higher cut scores than they intend, whereas RP50 might provide more 
flexibility in lower ranges of the score scale. In some cases the preferred RP value from a policy 
perspective may be at odds with the RP value that is best from a methodological perspective. In 
these cases, states must decide which RP criterion is the most appropriate choice for standard 
setting. In the end, the best decision from a standard setting methodology point of view may be 
to select the ordering information and criterion that ensures that adequate numbers of items are 
available in the OIB for the full range of examinee proficiency. 


Modified Angoff Standard Setting Method. Even though the modified Angoff cognitive- 
judgmental task has been famously deemed a complex, “fatally flawed” cognitive task (Shepard, 
Glaser, Linn, & Bohrnstedt, 1993), it, along with the Bookmark Procedure, account for the major- 
ity of standard settings for state assessment programs over the last 15 years. Use of the modified 
Angoff method for educational achievement tests can be traced to the 1970s. The modified Angoff 
method typically is implemented in two or three rounds of judgments and discussion,’ where 
panelists discuss item response demands and target examinees and make judgments about every 
item in a test book. The key components in a modified Angoff standard setting are the test items, 
typically presented in the test books that examinees use, and conceptualizations of target students. 
In each round, the panelists’ cognitive-judgmental task is to estimate the percentage of target 
examinees (e.g., students who are just barely Proficient) they expect to respond successfully to a 
multiple-choice item or at a score level on a constructed-response item. Typically, panelists view 
impact data (e.g., percentages of students at or above the Proficient level) to inform their final 
judgments. To calculate cut scores for each round, the average expected percentage for each item 
across all panelists is calculated, the averages summed and translated into a total test cut score. 


“In the original (unmodified) Angoff method, standard setting panelists judged simply whether a target examinee 
would respond correctly or incorrectly to a multiple-choice item. 
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Full descriptions of materials, training, and procedures for the modified Angoff method appear 
in several texts (Cizek & Bunch, 2007; Zieky, Perie, & Livingston, 2008). 


Challenges in Using the Modified Angoff Method for AA-MAS. One main challenge 
should be considered before implementing the modified Angoff method to set cut scores for AA- 
MAS. As with the Bookmark Procedure, this challenge must be addressed in using the modified 
Angoff method for any achievement test. However, it requires special consideration for AA-MAS. 


Challenge I: Probability judgments. People are notoriously inaccurate in making probability 
judgments. The amount of empirical research and the range of contexts in which this observation 
has been documented are impressive. People can be trained to estimate probabilities “moderately 
well” (Nickerson, 2004, p. 433), but they are susceptible to judgmental biases and are prone to 
making errors when judging the probability of an occurrence (Nickerson, 2004, chap. 11; Plous, 
1993, p. 144). Although probability type judgments are embedded in the Bookmark judgmental 
task through the response probability criterion, probability judgments in the form of percentages 
are the cognitive-judgmental task for the modified Angoff method. We acknowledge that the 
modified Angoff method continues to be used to produce appropriate and acceptable standards. 
In addition, standard setting panels have been shown to produce cut scores from the modified 
Angoff method which are not that different from cut scores produced using the Bookmark 
Procedure (cf. Karantonis & Sireci, 2006, p. 8), indicating that standard setting panels produce 
appropriate and acceptable cut scores using the modified Angoff method despite its complexity. 

Probability judgments, however, may be more difficult for AA-MAS, at least for now. The 
field is struggling to define the target examinee population, so it may be even more difficult for 
standard setting panelists to estimate percentages in the modified Angoff method, especially if 
the target examinee population is not clearly defined and understood. Training can help improve 
probability judgments (Nickerson, 2004, p. 433). Training panelists to understand clearly the 
target examinee population is a special requirement for setting standards for AA-MAS using the 
modified Angoff method. 


Challenge 2: The number of judgments required. The judgmental task associated with the 
modified Angoff method asks panelists to consider target examinees and estimate the percentage 
that would answer a multiple-choice item correctly or achieve a score level for a constructed- 
response item. Although this is not a challenge unique to assessments of modified achievement 
standard, it does warrant consideration. Using a 30-item AA-MAS and only two cut scores 
to produce Advanced, Proficient, and Basic levels (the minimum requirement) to illustrate, 
panelists would have to estimate the probability of successful responses for 30 items for the 
Basic/Proficient cut score and 30 for the Proficient/Advanced cut score. Panelists would have to 
make 60 probability judgments—judgments that they probably find challenging—for each of two 
or three rounds of standard setting. 


Standard Setting for AA-MAS Using Collections of Evidence. At least one state so far 
has designed an AA-MAS using portfolios containing student work as evidence of achievement. 
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Assessments based on collections of evidence are most compatible with standard setting methods 
that base judgments on student work rather than on test items. Here we address two of the most 
widely used of those methods. 


Body of Work and Performance Profile Sorting Methods. Some states may base their 
designs for AA-MAS, not on conventional grade-level assessments, but on the design approaches 
for their alternate assessments based on alternate achievement standards. Eleven of 31 states 
with information available on their Web sites use portfolio designs for their alternate assessments 
(Ferrara et al., 2009, Table 1 and surrounding text). Portfolio assessments contain collections of 
student academic work, video and audio recordings of them as they perform academic tasks, and 
other evidence of their current performance in relation to extended grade-level content standards. 
Portfolios, or collections of student work, have been used to guide instruction effectively, but 
with limited psychometric rigor, for assessing students with significant cognitive disabilities and 
for assessing mathematics and writing achievement of all students in selected grades in Vermont 
(Koretz, Stecher, Klein, & McCaffrey, 1994). Portfolio assessments lend themselves to standard 
setting methods that focus on student work or ratings of the quality of the work, rather than on 
test items. 

Two of the most widely known of these methods are the Body of Work and Performance 
Profile sorting methods (see Cizek & Bunch, 2007; Hambleton & Pitoniak, 2006). In the Body of 
Work method, standard setting panelists examine student work, typically contained in portfolios, 
that represents the range of possible total portfolio scores. They sort the portfolios in two different 
rounds into groups that correspond to the achievement levels for which standards are being set. 
Cut scores are calculated using logistic regression (simpler calculations are also possible) to 
identify portfolio scores that most clearly distinguish the scores of panelists’ groupings of student 
work into performance levels. Because preparing and organizing student work is such a large 
logistical task, and because sorting large numbers of portfolios can be arduous for panelists, some 
standard setters may prefer the Performance Profile sorting method (see Cizek & Bunch, 2007; 
Hambleton & Pitoniak, 2006). 

As in Body of Work, Performance Profile panelists sort evidence of student performance into 
achievement levels. In this case, score profiles are sorted, rather than student work. Specifically, 
panelists review visual or numerical displays of score combinations (i.e., score profiles) of student 
work (e.g., accuracy scores on a 4- or 6-point scale for several pieces of student work),° sorted 
from lowest to highest, and draw lines to determine cut scores for achievement levels. Because 
some total test scores can be achieved by several score profiles, panelists examine score profiles 
for each total test score. Cut scores are calculated as described for the Body of Work method. 
Although Performance Profile sorting reduces the logistical burden of the Body of Work method, 
the number of score profiles that panelists must consider and sort can be large. The subtlety and 
complexity of judgments required to distinguish profiles is challenging, for example, whether a 
score profile of (6, 6, 7) should be sorted into the same achievement level as a score profile of (4, 
5, 10). 


5Scoring scales for some portfolio assessments can be as high as 10 per collection of evidence, as when evidence is 
scored once for accuracy on a 1- to 5-point scale and once for support on a 1- to 5-point scale. 
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Challenges in using the Body of Work method for AA-MAS. We already have alluded to 
the significant logistical challenge posed by the Body of Work method. In addition, it has been 
common to discover that almost no student work can be located for the lower score levels for 
alternate assessments, because so few students receive low scores. For some alternate assessments, 
it has been necessary to simulate performance at the lower score levels, which may degrade the 
quality of student portfolios. We do not know yet whether this finding or the opposite—too little 
evidence at the top end of the scale—will occur for AA-MAS. 


Challenges in using the Performance Profile sorting method for AA-MAS. In Performance 
Profile sorting, the logistical challenge is reduced because only small numbers of exemplar 
student work collections are necessary. However, the cognitive-judgmental burden and complexity 
increase significantly for standard setting panelists. For example, a simple portfolio that includes 
three pieces of evidence, each scored on a 1- to 4-point scale, produces 43 (or 64) score profiles 
for panelists to examine and sort into achievement levels. As we illustrated earlier, some profiles 
cannot be sorted easily. For example, because panelists know that requirements of the three 
pieces of evidence may not be equivalent, they may struggle with deciding whether to sort a 
score profile of (4, 3, 4) into the same achievement level as the score profile (4, 3, 2) or with 
the score profile (4, 4, 3). Making this discrimination may be particularly difficult for AA-MAS 
because the target examinees are capable of challenging grade-level work and display uneven 
performance in content areas because of their disabilities. 

Standard setting designers can devise methods for reducing the number of profiles to be 
examined. For example, there is little question that profiles of (1, 1, 1) and any combination of 
two Is and a 2 are going to be sorted into the lowest achievement level, so reviewing them may 
not be necessary. States may also provide panelists direction on a strategy that emulates the state’s 
perception of target study by achievement level. For example, the state may direct the panelists 
to give greater “weight” to certain standards as reflective of the blueprints or, as in the case of 
alternate assessments, to higher levels of independence. In any case, managing the complexity 
of judging the performances of students with disabilities may remain a challenge for standard 
setting panelists. 


VALIDATION STUDIES 


Evidence to support the validity of the interpretation of cut scores, performance level descriptions, 
and performance standards is conceptually complex and requires significant sustained effort. 
Collecting validation evidence begins with design, development, and implementation of a test and 
carries through to the implementation of the cut scores themselves. In the discussion of standard 
setting methods for AA-MAS, we addressed Kane’s (2001) concern that standard setting methods 
should be consistent with the design and type of test. We suggested two designs for AA-MAS and 
corresponding standard setting methods. The rationale for the method selected should be clearly 
stated in the standard setting design. 

In this section, we touch on external checks that a state can reasonably undertake in collecting 
evidence of validity. We discuss evidence based on relations to other measures, as required 
in peer review guidance for AA-MAS (USDE, 2007b) and the Standards for Educational and 
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Psychological Testing (American Educational Research Association, American Psychological 
Association, & National Council on Measurement in Education, 1999). Evidence of relations with 
other measures would compare the results from the application of cut scores on AA-MAS to other 
relevant results. Students’ classification into achievement levels can be compared to performance 
on other tests, in classroom assessments, and other meaningful measures. It is important to 
articulate the intended interpretations of performance on AA-MAS and how performance in the 
achievement levels should compare to levels of performance on the grade-level assessments. 


Contrasting Groups Survey 


A second standard setting method sometimes is used as evidence of external validation of perfor- 
mance standards. However, it is common to find that different standard setting methods produce 
different results. Assuming both methods are well implemented, this leaves a state in a predica- 
ment, having two sets of recommended cut scores from which to produce one set of cut scores. 
If the purpose of the second method is to validate, it is not helpful or useful to obtain a different 
set of results. 

Instead of implementing a second standard setting workshop, the Contrasting Groups method 
may be implemented as a survey. Using the survey, teachers classify each of their students into 
an achievement level using the PLDs for an assessment. In using the Contrasting Groups method 
for validation rather than setting cut scores, the purpose of the survey would be to see how 
well teacher judgments of student performance align student performance on the assessment 
specifically, the performance levels in which their scores place them. New cut scores would not be 
derived from the survey results. Instead, classification consistency analyses would be conducted 
to compare teacher classifications of student performance and classification of students based on 
the assessment. 

The survey could be implemented either through a paper-and-pencil mailed survey or a 
Web-based approach. The challenge of implementing the Contrasting Groups approach as a 
survey is ensuring that all participants consider the PLDs before classifying students into an 
achievement level. If the survey is implemented through the Web, teachers simply can click 
on a PLD for each student. The amount of time spent on each screen could be captured and 
analyzed to ascertain if participants actually spent time analyzing the PLDs. With both ap- 
proaches, participants can be asked how rigorously they considered the PLDs before classify- 
ing students. Unfortunately, there is no way to guarantee that the participants study the PLDs 
before classifying students. To encourage thoughtful responding, states can administer the sur- 
vey at other test-related workshops, such as Content and Bias Review, where a diverse group 
of panelists is available. A small portion of time could be devoted to the survey. The sur- 
vey probably cannot be administered during a standard setting workshop. In many cases, state 
departments or boards of education adjust cut scores in light of policy considerations follow- 
ing a standard setting workshop so that final cut scores may not be available until well after 
the standard setting workshop. It is desirable that an independent group provide the validation 
information. 

If the results from the survey and the test-based classifications do not correspond closely, the 
state will need to examine the reasons for this. It may uncover a disconnect between the PLDs 
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and the test performance. It may also reveal that teachers tend to over- or underestimate student 
performance in relation to the performance levels and PLDs. 


Comparison of Results from AA-MAS and Grade-Level Assessments 


Another approach to collecting validation evidence could involve two groups of students taking 
both AA-MAS and grade-level assessments. One group would be a diverse sample of examinees 
who are eligible for the AA-MAS (the target examinee group), the other a diverse sample of 
students who are eligible for the grade-level assessments (the comparison group). We would 
expect the average scores of the two groups to be considerably different on both assessments. 
Further, we know from other studies (e.g., Fincher, 2008) that large numbers of the target examinee 
group will perform in the lowest achievement level of the grade-level assessment. Likewise, we 
expect large numbers of the comparison group students to perform at the highest achievement 
level of the AA-MAS. If the PLDs for the AA-MAS are developed in relation to the PLDs for the 
grade-level assessments, we would expect student performance to provide support. For example, 
if Proficient on AA-MAS is intended to be similar to Basic on the grade-level assessments, we 
would expect that students classified as Basic on the grade-level assessment would, in general, 
be classified as Proficient or Advanced on the AA-MAS. 

Of course, grade-level assessments are not appropriate for students who are appropriately 
eligible AA-MAS. Other approaches that examine response demands of items that define each 
cut score (e.g., Ferrara et al., 2007) rather than require administration of items could provide 
useful validation evidence. In an item response demands study, the academic knowledge and 
skill demands of items that define Proficient performance on the AA-MAS could be compared 
to the knowledge and skill demands of items that define a corresponding level on the grade-level 
assessments. 


CONCLUSION 


As we indicated at the beginning of this article, requirements for developing and validating 
educational achievement tests apply to AA-MAS with the same rigor and importance that they 
apply to grade-level assessments. However, AA-MAS are unique because the target examinee 
population is unique and relatively unfamiliar to designers of large-scale educational achievement 
tests. In addition, AA-MAS are new enough that information is still emerging about the most 
effective and affordable approaches to assessing the target examinees. 

Some points are clear. It is crucial to define and describe the target examinee population. This 
has become a routine process for grade-level assessments. The field has made progress in defining 
the target examinee population for alternate assessments for students with significant cognitive 
disabilities. However, progress in defining the target examinee population for AA-MAS is only 
beginning. It is standard practice to develop PLDs prior to standard setting and, in fact, early 
in the test design and development process. This practice also applies to AA-MAS, although 
it will continue to be particularly challenging to define the PLDs until we are able to define 
the target examinee population clearly. Finally, it is important to consider the challenges we 
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describe to implementing conventional standard setting ‘methods for AA-MAS. For AA-MAS, 
both conventional and innovative practices are required. 
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Since Federal regulations have given states the option to implement alternate assessments based on 
modified academic achievement standards (AA-MAS) as part of their accountability systems for a 
small group of students with disabilities, a number of states have made decisions about whether or 
not to develop and implement such an assessment. State-level directors of assessment and directors 
of special education were surveyed about their state’s decisions on implementing AA-MAS. Im- 
provements in accessibility and appropriateness were reasons given for choosing to implement an 
AA-MAS, whereas lack of resources and guidance were identified as barriers. This article presents 
the findings from a survey on 22 states’ decisions concerning implementation of AA-MAS. 


BACKGROUND 


The purpose of this article is to present states’ perspectives on alternate assessments based on 
modified achievement standards (AA-MAS) and to investigate the reasons the states have chosen 
to or not to implement them. No Child Left Behind (NCLB), in conjunction with the Individuals 
with Disabilities Education Act, has emphasized the use of academic content standards for 
instruction and large-scale assessment for all students, including those with disabilities. When 
the NCLB regulations were finalized in 2007, they gave states the flexibility to develop AA-MAS. 
According to the regulations, these assessments are intended to be used with a small group of 
students with disabilities whose progress may be significant but who are unlikely to achieve 
grade-level proficiency in the same time frame as other students. Students who participate in this 
assessment may be from any disability category, but they must have access to grade-level content 
(U.S. Department of Education, 2007a). 

The federal regulations require that these assessments cover the same grade-level content 
as the general assessment. They are to be based on modified academic achievement standards, 
or expectations of mastery that have been modified. These regulations do not require states to 
implement an AA-MAS, but according to Filbin (2008), the flexibility does provide an opportunity 
for them “to conceptualize and create a framework for teaching, learning, and assessment that 
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supports high expectations for all students with disabilities while providing educators and parents 
with a deeper understanding of how these students demonstrate their knowledge” (p. 2). Although 
these AA-MAS are not required, a number of states have begun to implement them, while a number 
are still investigating or have chosen not to implement an AA-MAS at this time. 


SURVEY OF STATE DIRECTORS 


Two surveys were created to collect information from each state level director of assessment and 
each director of special education in the United States. One survey was designed for states that 
have chosen to implement an AA-MAS, and the second was created for states who have chosen 
not to do so at this time. The surveys were available for 6 weeks and asked the two directors, or 
designee from each state, to consider both barriers to and implications of AA-MAS. The total 
number of responses to the survey invitation was 24. The text of these two surveys along with the 
response count for each question are included in the appendix. 

Of the responses received to the survey, 22 (44%) states were represented. Table | presents 
the number of responses to each survey by the respondent’s department association. There were 
responses from 11 states to the survey for states with an AA-MAS and 11 to the survey for states 
without an AA-MAS. One state had two individuals from the assessment division complete the 
appropriate survey. In only 1 state did both the assessment and special education department both 
respond. 

Respondents that completed the survey indicating they were implementing an AA-MAS are 
at various stages of implementation. Four of the states represented were among the six states to 
have submitted evidence for Title I Peer Review of their AA-MAS in March of 2008. A number 
of grades and content areas were assessed by the four states using AA-MAS (see Table 2). 

The remaining eight states indicated that they were still in the development process or had not 
yet administered an AA-MAS. Of those states, one indicated the development process was cur- 
rently on hold while participation guidelines, standards-based IEP guidelines, technical assistance 
documents, and training procedures are being developed. 


WHY IMPLEMENT AN AA-MAS? 


Prior to the release of regulations on AA-MAS, the eligible group of students with disabilities 
was required to take either a grade-level assessment, with or without accommodations, or an 


TABLE 1 
Number of Responses to Each Survey by Respondent's Department 

b PAP Pp Snide ee 8 ———————— 

With an Without an 
Department AA-MAS AA-MAS 
le a Na eal EL RS A IS le Rh 
Assessment Department 8 7 
Special Education Department 4 > 


a aliynipiehe Yet eee Ore 
Note. AA-MAS = alternate assessments based on modified academic achieve- 
ment standards (AA-MAS). 
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TABLE 2 
Grades and Participants Included in the AA-MAS of the Four States That Have Implemented 

English/ Social 
Grade Reading Mathematics Writing _ Language Arts Science Science 
ee ee ee ee SS eS 
Grade 3 3 3 1 
Grade 4 5 3 1 Z 
Grade 5 3 5 1 1 1 1 
Grade 6 3 3 1 
Grade 7 3 3 1 2 
Grade 8 5 3 1 1 1 1 
High school 2 4 1 8 4 3 


nnn n ee CUES EEESEEEESEEESSSSS EEE 


Note. AA-MAS = alternate assessments based on modified academic achievement standards (AA-MAS). 


alternate assessment based on alternate academic achievement standards (AA-AAS). According 
to Non-Regulatory Guidance from the U.S. Department of Education (2007b), “neither of these 
options provides an accurate assessment of what these students know and can do” (p. 8). Because 
a grade-level assessment is likely too difficult, it does not provide useful data about the abilities of 
these students. The Non-Regulatory Guidance continues to explain that AA-AAS are not intended 
to assess a student’s achievement across the full range of grade-level content and are therefore 
unlikely to provide the information needed to help these students progress toward grade-level 
achievement. When asked why their state chose to develop an AA-MAS, many of the responses 
mirrored the language in the Non-Regulatory Guidance. For example, one state responded it 
was developing an AA-MAS because there is a population of students for whom the general 
education assessment is not accessible and for whom the AA-AAS is not appropriate. The terms 
“accessible” and “appropriate” were commonly used by respondents to explain why their state 
was implementing an AA-MAS. A respondent to this survey expressed their state implemented 
the AA-MAS “to provide for an appropriate, valid, and reliable assessment for students with 
persistent learning difficulties who could not appropriately access the state’s standard test with 
accommodations.” Another state added, “we believe that an AA-MAS will have a positive impact 
on instruction of students with disabilities.” 

It is not surprising that there is quite a bit of overlap between responses addressing the ques- 
tion of why a state chose to implement an AA-MAS and their responses about advantages of an 
AA-MAS. In addition to the terms “accessible” and “appropriate,” that were used frequently in 
responding to the previous question, respondents also noted that AA-MAS may provide oppor- 
tunities for students to demonstrate knowledge and skills, and for these students to experience 
success. Some respondents also noted that scores from an AA-MAS will provide more meaning- 
ful data to teachers about student performance. There is a belief that the AA-MAS may provide 
information to teachers about student performance toward grade-level standards, rather than just 
indicating whether students have achieved the standard or not. 

Some of the respondents from states that have chosen not to implement an AA-MAS still 
identified advantages for such an assessment. For example, these respondents noted that an AA- 
MAS has the potential to meet student needs by increasing access for students with disabilities. 
They also acknowledged the potential for the new assessment to provide more information on 
instructional weaknesses and increase flexibility for reporting proficiency under the NCLB. The 
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majority of respondents whose states have chosen not to implement an AA-MAS at this time, 
however, did not identify advantages. 


DISADVANTAGES AND BARRIERS TO IMPLEMENTATION 


When asked to consider the disadvantages of implementing an AA-MAS, the responses of 
representatives of states who have implemented an AA-MAS can be grouped into three primary 
categories: (a) federal requirements, (b) effectiveness, and (c) resource availability. There were 
three respondents who cited issues with the clarity of the requirements. Specifically, a respondent 
said that from the state perspective the lack of federal guidance upfront combined with the 
subjective and murky peer review process on the back end create the greatest disadvantages to 
implementing AA-MAS. A respondent from a state that has implemented an AA-MAS cited 
“receiving Peer Review approval” as a perceived barrier to implementing an AA-MAS. There 
were three respondents that seemed to express disagreement with some of the requirements 
of using an AA-MAS. One respondent would have favored formative assessments be used for 
the AA-MAS instead of summative assessments. The other two respondents who believed the 
guidance provided or regulations were a disadvantage focused their responses on participation 
criteria and the limits on the number of students who can take an AA-MAS and be counted as 
proficient in adequate yearly progress (AYP) calculations. The final regulations on AA-MAS 
allow States to count the proficient and advanced scores of students who take AA-MAS toward 
AYP calculations if the number of those proficient and advanced scores does not exceed 2% of 
all students in the grades assessed. 

Concerns about ineffectiveness were also cited as a possible disadvantage by states that have 
chosen to create an AA-MAS. A few of the states that had not yet implemented an AA-MAS 
were unsure if it would have the impact that they hope. One of respondents from a state that has 
chosen to create an AA-MAS, but has not yet implemented it, said, “It may not help.” Another 
such respondent was concerned that an AA-MAS based on the current regulations and guidelines 
may not “yield useful information” about student knowledge and skills. 

Resource availability, including time and money, was commonly listed as a disadvantage to 
implementing an AA-MAS. Financially, the perceived costs varied significantly among those 
who responded. In one case, a federal grant covered the costs. At the other extreme one state 
estimated it will cost “at least $6.6 million a year once all levels are phased in.” One respondent 
noted that there are also human capital costs at the state department of education and the local 
education agencies. In that state, adding the AA-MAS requires staff working with assessments 
for students with disabilities to be spread more thinly to train teachers and administrators. A 
respondent from a state that has chosen not to implement an AA-MAS provided a summary of 
perceived costs: 


Even if the regular assessment itself is modified, this is equivalent to developing and administering 
and scoring a completely separate assessment. We’d need to conduct research on comparability, 
validity, reliability, impact, use of accommodations, etc.—all of that costs quite a bit. We’d need to 
conduct a standard setting, and do an alignment study. The training and professional development 
would need to be accompanied by lots of professional development about much more than just the 
test administration and interpretation. Attention needs to be paid to the implications for IEP teams, 
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teachers’ instructional methods and focus, etc. If we aren’t doing anything more than giving a new 
test and we can’t afford to do the professional development piece across the state, we likely won’t 
see any impact from this assessment on student achievement. 


Just as there was significant overlap between the perceived advantages of implementing 
an AA-MAS and the reasons respondents gave that their state had done so, there was much 
commonality between perceived disadvantages or barriers and the reasons respondents gave for 
not implementing an AA-MAS. The federal requirements, questions about effectiveness, and 
the drain on resources were the primary barriers to implementing an AA-MAS. One of the 
respondents from states where a decision has been made to not develop an AA-MAS raised 
questions when asked the disadvantages of an AA-MAS, wondering if this is “just seeking a way 
to ‘call’ some students ‘proficient’ on grade-level content when they really are not, just so that 
the schools have a better chance of making AYP for students with disabilities?” The respondent 
continued, “If that is the reason (and I believe most of the time it is), that by itself is not a good 
reason to develop a different assessment.” Another respondent from a state without an AA-MAS 
believed that it further complicates a number of things, including data comparability and the 
principle of common expectations for all. 


EFFECTS ON CURRICULUM AND INSTRUCTION 


Both sets of respondents were asked to consider the effect AA-MAS would have on curriculum 
and instruction. Respondents from states that have chosen to develop an AA-MAS generally 
perceived that a positive influence on curriculum, instruction, and the general assessment is likely. 
One of the effects respondents anticipate is a positive impact on the amount of time students with 
disabilities receive instruction on grade-level content. A representative of one state said, “We are 
receiving feedback from teachers that the AA-MAS is further supporting the use of standards- 
based education in the state.” A respondent from a state with an AA-MAS reported there is some 
evidence that the curriculum for students with disabilities is becoming more standards-based in 
some locations. 

The majority of responses from representatives for states that have chosen not to develop an 
AA-MAS and some respondents from states with an AA-MAS did indicate that the AA-MAS 
should not change what is taught because it is based on the same content standards as the general 
assessment. The curriculum should be the same, so instruction should be provided on the content 
of the grade in which the child is enrolled. 


EFFECTS ON THE GENERAL ASSESSMENT 


Representatives from states that have chosen to implement an AA-MAS were asked to consider 
the effects it may have on the general assessment. Some of the respondents did not believe that 
there would be any effect. Of those that perceived an influence, their responses were varied. One 
respondent cited that the AA-MAS will decrease the number of students with disabilities taking 
the general assessment, whereas another predicted it may influence accommodation guidelines. 
The most commonly cited effect of the AA-MAS on the general assessment from representatives 
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of states that have developed one was that the process may lead to greater enhancements to 
the general assessment and increased efforts to offer universally designed assessments for all 
students. 


CONCLUSIONS 


Since states have been given the added flexibility to implement AA-MAS as part of their ac- 
countability systems, a number of states have made decisions about whether or not to develop 
and implement such an assessment. The states that have chosen to utilize the flexibility are in 
various stages of implementation. States’ perspectives on alternate assessments based on mod- 
ified achievement standards are varied. Overall, it appears that states want all students to have 
an opportunity to demonstrate their skills and have access to the general curriculum. However, 
states realize that developing measurement tools that yield reliable and valid scores for making 
inferences is demanding and costly work. Time, money, and guidance are required to sufficiently 
develop an AA-MAS. Consistent guidance regarding the technical specifications for developing 
and implementing an AA-MAS is desired. States’ AA-MAS are evolving. As more states grapple 
with decisions about offering an AA-MAS, more states also approach and go through the peer re- 
view process. These advances will likely add new perspectives to decisions about implementation 
of alternate assessments based on modified academic achievement standards. 
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APPENDIX 


TABLE A1 
Questions Included in the Survey of States With an Alternate Assessments Based on Modified Academic 
Achievement Standards (AA-MAS) 


i ee EEEEEEE EEE 


Survey Question Response Count 


on eee eee 





1. What is your name, title, and state? 12 

2. What grades and content areas are included in your AA-MAS? 12 

3. Why did your state develop an AA-MAS? 12 

4. What percent of students with disabilities in your state are performing 11 
below proficiency in reading and mathematics? 

5. What are the advantages of the AA-MAS? 12 

6. What are the disadvantages of the AA-MAS? 11 

7. Describe the students who will take your AA-MAS. 12 

8. How will proficiency shown via the AA-MAS differ from proficiency 11 

shown through the general assessment? 

9. How will the AA-MAS affect the instruction that teachers deliver? 11 
10. How will the AA-MAS affect the curriculum that teachers deliver? 10 
11. What are the perceived costs associated with implementing the 12 

AA-MAS? 
12. What are the perceived barriers to implementing the AA-MAS? 12 
13. What effect, if any, will an AA-MAS have on your general assessment? 12 
TABLE A2 


Questions Included in the Survey of States Without an Alternate Assessments Based on Modified Academic 
Achievement Standards (AA-MAS) 





Survey Question Response Count 
1. What is your name, title, and state? 12 
2. Why did your state decide not to develop an AA-MAS at this time? 12 
3. What percent of students with disabilities in your state are performing 10 

below proficiency in reading and mathematics? 
4. What are the advantages of an AA-MAS? 11 
5. What are the disadvantages of an AA-MAS? 12 
6. Describe the students who would take an AA-MAS in your state. 11 
7. How are these students assessed in your state? 11 
8. How would an AA-MAS affect the instruction that teachers deliver? 12 
9. How would an AA-MAS affect the curriculum that teachers deliver? 11 
10. What were the perceived costs associated with implementing an 12 
AA-MAS? 


11. What were the perceived barriers to implementing an AA-MAS? 10 
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In this commentary on the Peabody Journal of Education special edition, the author addresses 
implications of the contributing articles to three central domains of interest to states engaged in or 
considering the development of an alternate assessment on modified academic achievement standards: 
(a) identifying an eligible student population, (b) ensuring eligible students are provided access to 
appropriate instruction, and (c) developing more accessible test items. 


The April 9, 2007, joint Title I Individuals with Disabilities Education Act regulations which 
authorized States to develop and implement alternate assessments based on modified academic 
achievement standards (AA-MAS) left largely to states the task of determining how the population 
of students eligible to take the assessment would be identified and defined. Whereas for several 
years earlier, most states had little difficulty identifying the population of students with the “most 
significant” cognitive disabilities pursuant to the December 9, 2003, Title I rule authorizing the 
implementation of alternate assessments on alternate academic achievement standards, the 2007 
joint regulation necessitated substantially more time and resources to conduct investigations 
into the learning experiences and characteristics of the target population. As Egan, Ferrara, 
Schneider, and Barton (2009/this issue) comment, the task of identifying students eligible for 
an alternate assessment based on modified academic achievement standards has become an 
“evolving, inferential, conceptual, and empirical enterprise”. The 2007 joint regulation described 
the eligible population as: 


A small group of students with disabilities whose progress is such that, even after receiving appro- 
priate instruction, including special education and related services designed to address the students’ 
individual needs, the students’ individualized education program (IEP) teams are reasonably certain 
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that the students will not achieve grade-level proficiency within the year covered by the students’ 
IEPs. (U.S. Department of Education, 2007, p. 17748) 


Inherent in the description of the eligible target population was the requirement that the 
barrier to attaining grade-level proficiency among this small group of students must not be 
attributable to lack of opportunity to learn grade-level material, but rather to the limitations im- 
posed by the disability itself. The regulation made clear that eligible students were to be “students 
whose disability [italics added] has precluded them from achieving grade-level proficiency and 
whose progress is such that they will not reach grade-level achievement standards in the same 
time frame as other students” (p. 17748). 

Also implicit in the regulation was the intent of the Department to remedy a problem created 
by the absence of an appropriate state assessment option for this group of students: 


Currently, these students must take either a grade-level assessment or an alternate assessment based on 
alternate academic achievement standards. Neither of these options provides an accurate assessment 
of what these students know and can do. A grade-level assessment is too difficult and, therefore, does 
not provide data about a student’s abilities or information that would be helpful to guide instruction. 
An alternate assessment based on alternate academic achievement standards is too easy and is not 
intended to assess a student’s achievement across the full range of grade-level content. Such an 
assessment, therefore, would not provide teachers and parents with information to help these students 
progress toward grade-level achievement. (U.S. Department of Education, 2007, p. 17748) 


From its inception, one of the aims of the AA-MAS has been the quest for more accurate 
information about the knowledge and skills the identified students have acquired, by which to 
guide their future instruction. One of the most critical tasks facing states electing to develop an 
AA-MAS has been the identification of an eligible population of students—students who, despite 
appropriate instruction, are unable to demonstrate their acquired knowledge and skills on the 
available state assessments. 

To ensure that ineligible students (such as students whose inability to demonstrate proficiency 
on a general assessment is solely attributable to lack of opportunity to learn (OTL) appropriate 
grade-level material) are not included among students for whom the AA-MAS is deemed appro- 
priate, it behooves a state to undertake a comprehensive examination of the instructional access 
provided to the target population on whose behalf the assessment is being developed. To accom- 
plish these investigations, many states have undertaken studies of the alignment of instruction 
provided to the target population to the state’s grade-level academic content standards. 

Complicating the development of the AA-MAS is the fact that the persistently low test 
performance among eligible students should also not have been solely because of the need for 
more accessible test items per se—as the AA-MAS should contain test items which are not 
merely more accessible, but are additionally of reduced difficulty. It is important also that, when 
investigating the needs of students with disabilities who have been persistently low performing, 
consideration be given to the likelihood that a portion of this population may in fact be quite 
able to demonstrate proficiency on an alternate assessment on grade-level academic achievement 
standards (of which only a few states have peer-reviewed and approved versions). 

At the present time the U.S. Department of Education supports 31 states by means of ei- 
ther General Supervision Enhancement Grants funded through the Office of Special Education 
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Programs or Enhanced Assessment Grants funded through the Office of Elementary and Sec- 
ondary Education, in the development of alternate assessments. Most of these funded states are 
actively engaged in investigations into the group of special education students unable to demon- 
strate proficiency on either a general or alternate assessment. The fruits of such investigations, 
carried out for the dual purposes of test development and implementation, are certain to be en- 
hanced by the information provided in the articles contained this Peabody Journal of Education 
special issue. In the commentary next, I address some of the salient points made by the con- 
tributing authors within three central domains of interest to states developing an AA-MAS: (a) 
identification of the eligible student population, (b) ensuring access to appropriate instruction, 
and (c) developing a more accessible assessment. 


IDENTIFICATION OF THE ELIGIBLE STUDENT POPULATION 


Palmer (2009/this issue) presents interesting findings from a survey completed by several states 
either considering or engaged in the development of an AA-MAS. It was gratifying to read that 
the respondent state motivations for developing this assessment centered on the desire to create 
a test that would better inform instruction of participating students. Because this assessment 
option was originally offered as a flexibility to states in the context of NCLB accountability, the 
concern some states have expressed—that the assessment “may not help” accountability, at least 
in the short term—suggests that at least some states realize that many eligible students will still 
not, at least initially, perform much better on any test aligned to grades-level content standards. 
In light of legitimate concerns about the costs of funding the development of the AA-MAS, 
it is understandable that the potential benefit to a state in terms of accountability may not be 
sufficient to justify the cost of development. It was gratifying to read in Palmer’s account that 
most respondent states justified the substantial cost of implementing this assessment by reference 
to the potential benefit it provided to the target group. 

Lazarus and Thurlow (2009/this issue) also provide information on early state efforts to develop 
the AA-MAS. They write that most of the states in a position to develop these assessments 
early were those that had previously administered “‘out-of-level” assessments, a practice later 
disallowed for the purposes of documenting participation and performance on state assessments 
under NCLB. Lazarus and Thurlow raise an interesting set of questions that will undoubtedly 
prove useful in the course of continued tracking of the national picture as more states begin 
to develop these assessments. Although as the authors comment, most states had attained high 
levels of assessment participation without implementing an AA-MAS, it is important to consider 
that IEP Teams have had, until the advent of the AA-MAS limited options to reduce expected 
achievement standards for students with disabilities. 

The question of how the new assessment will contribute to improving performance over time 
is a more important consideration when weighing the advantages of implementation. If states 
implement the AA-MAS, as Palmer’s (2009/this issue) results suggest, predominantly to provide 
better information to inform instructional interventions and accelerate learning, the academic 
benefit to and growth over time for participating students will be an outcome of great interest. 

The task of establishing a coherent and inclusive assessment system, with the capacity to 
accommodate learning differences while measuring appropriately aligned content knowledge 
and skills is certainly the ultimate goal of every state Department of Education’s assessment 
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division. However, such an inclusive system can only be the result of a thorough understanding of 
the issue of inclusion—both for purposes of instruction and assessment. Because, by definition, 
students for whom the AA-MAS is required are students for whom grade-level achievement 
standards are inappropriate, the first step in developing a defensible alternate assessment on 
modified academic achievement standards must be a thorough investigation of the state’s history 
of inclusion of eligible students in the general education curriculum, and the actual results of 
such inclusion. 

Zigmond and Kloo (2009/this issue) present a review of historical and legislative decisions 
reflecting the inclusion of students with disabilities in the curriculum and in general assessments. 
They remind us that as early as the 1994 Reauthorization of the 1965 Elementary and Secondary 
Education Act, otherwise known as the Improving America’s Schools Act, each state receiving 
Title I funds was required to ensure the participation of all students in state assessments. Zigmond 
and Kloo describe this period as an era in which exclusion of special education students from 
large-scale testing as well as “grade-level performance” was considered a humane response to 
students with disabilities. The NCLB amendment provisions extended these provisions, requiring 
that students with disabilities be held to the same academic and performance standards as everyone 
else, and prohibiting their exclusion from the accountability system. Although the years between 
2001 and 2006 witnessed important progress by students with disabilities on state assessments, 
nevertheless during the 2007-08 school year it is sobering to note that only about 30% of students 
with IEPs nationwide scored at a proficient level either in reading or math (Thurlow, Bremer, & 
Albus, 2008). 

Zigmond and Kloo take up a number of questions currently being pondered in school divisions 
around the country: To what extent have students with disabilities really been exposed to the gen- 
eral education curriculum? To what extent might their performance problems reflect inadequate 
provision of accommodations, Universal Design of assessments, or the burden of trying to master 
an inordinate number of state content standards? The three articles written by Palmer, Zigmond 
and Kloo, and Lazarus and Thurlow promise to advance the direction of these discussions within 
the field. The problem of ensuring guidance and training to IEP teams that are expected to deter- 
mine student eligibility appropriately and consistently from year to year will soon fall to states. 

To fulfill these responsibilities effectively, IEP teams must be provided with guidelines by 
which to differentiate between a student who, because of the impact of a disability, will benefit 
from the greater accessibility and precision of measurement afforded by an AA-MAS from the 
student who, with more aligned instruction or more appropriate accommodations, could validly 
demonstrate the extent of his knowledge and skill on the general assessment. 


ENSURING ACCESS TO APPROPRIATE INSTRUCTION 


The majority of states engaged in developing an AA-MAS are drafting and refining guidelines 
for the identification of students whose disabilities are such that neither an assessment based 
upon grade-level achievement standards nor an assessment based upon alternate achievement 
standards will provide a fair and accurate picture of what these students know and can do. At the 
same time, it is incumbent upon implementing states to ensure that the target group of students 
with disabilities for whom these tests are being developed have been and continue to be provided 
the opportunity to learn the content covered on the assessment and to make progress, hopefully 


PERSPECTIVES ON STATE OF ALTERNATE ASSESSMENTS 589 


to “catch up” to their age-peers to the extent that they may become eligible to take the state’s 
regular assessments against more challenging achievement standards. 

However, the target population, even with appropriate instruction, will by definition require 
more time to master grade-level material, and this pace of learning will preclude the students’ 
ability to attain proficiency by the end of the school year without increasing instructional time 
during the school year, or extending the academic calendar year. 

As Zigmond and Kloo (2009/this issue) remind us, the solution to this slower pace of learning 
has traditionally been the implementation of “elongated” courses with less demanding material, 
which ensure that students enrolled in them cover less material by the end of the academic 
year. But such solutions effectively reduce the opportunity to learn, unless ways can be found to 
increase instructional time simultaneously. Coverage of the full scope of the academic content 
standards with mastery of critical grade-level knowledge and skills will require new approaches 
to augment and intensify the instruction provided during the traditional school year. 

Lengthening the duration of education in years has also been a traditional approach to the 
dilemma of slower paced learning, but the historically low graduation rates among students 
with disabilities suggests new approaches to teaching children who learn at a slower pace are 
sorely needed. During the 2006-07 school year, more than 26% of students with disabilities 
were confirmed to have dropped out of school, whereas only about 56% graduated with a regular 
diploma (U.S. Department of Education, 2008). Instituting an “extended-time” approach within 
a traditional academic year only institutionalizes the practice of below-grade instruction and is 
incompatible with the expectation that a student take an end-of-year assessment aligned to the 
full scope of the state’s content standards. 

It is important to stress several adverse consequences of a decision to provide below-grade 
instruction to a student with a disability. Not only does below-grade instruction prevent the 
student’s access to the general education curriculum and deprive a child of the opportunity to 
learn grade-level content, but it also precludes any fair assessment of what the student knows and 
is able to do, as well as any valid interpretation of resulting scores. Moreover, the assessment of 
students on content they have been provided no opportunity to learn violates fair testing provisions 
cited in the Standards of Educational and Psychological Testing. A key requirement built into 
the joint Title I and Individuals with Disabilities Education Act regulation of 2007 attempts to 
safeguard the opportunity to learn—requiring that all students participating in AA-MASs have 
“access to the curriculum including instruction for the grade in which the student is enrolled” 
(U.S. Department of Education, 2007, p. 17778). 

As Common Core State Content Standards are developed, there is renewed hope among 
education leaders that general and special educators will leverage their resources to develop well- 
articulated and differentiated curricula aligned to the new standards for students with disabilities at 
all ability levels. In addition, these developments provide reason to hope that the critical learning 
progressions required to bridge the teaching of these standards within and between grades will be 
delineated, to ensure mastery of the material encompassed by the new content standards, within 
the course of every school year. 

Although the breadth of the content covered in the curriculum for students receiving special 
education is expected to align to what is covered in regular education—for students assessed 
against Modified Academic Achievement Standards, the depth of coverage may be reduced. In 
differentiating instruction, there may also be merit in prioritizing for mastery content standards 
that are reiterated from year to year, such that a student can revisit material that extends the 
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“essential learning” inherent in a content standard during subsequent years without departure 
from the overall instructional alignment to grade-level content standards in any given year. In 
addition, options for year-round or extended instructional services may prove to be a more 
productive option for students requiring extended learning time. 

Zigmond and Kloo (2009/this issue) make the important point that states face hazards when 
implementing an AA-MAS—if the wrong students are found eligible, teachers may “give up” on 
trying so hard to teach them. It is incumbent upon leaders in state Offices of Special Education 
to ensure that IEP teams are provided guidance to avoid the unintended consequences of inap- 
propriate eligibility decisions. Likewise, the implementation of the safeguards required in the 
regulation to ensure IEPs for such students are aligned with state-adopted content standards, and 
the progress of participating students is monitored, will help prevent unintended outcomes. 

As has been the case with students taking alternate assessments on alternate achievement 
standards, it is possible that some students found eligible for the AA-MAS will be held to a lower 
standard than they are capable of attaining. Nevertheless, by implementing standards-aligned 
instruction, by ensuring appropriate standard-setting on the new assessment, and by working 
toward a comprehensive and integrated assessment system inclusive of formative and benchmark 
assessments, states can assist participating students to benefit rather than suffer from assessment 
against a lower achievement standard. 

As Zigmond and Kloo (2009/this issue) mention, the field is only beginning to scratch the 
surface in providing curriculum-based assessments for use in tracking student progress in the 
curriculum. To realize the goal of tracking student progress optimally, states will need to not 
only provide aligned and integrated end-of-year assessments but also institute aligned formative 
assessments for teachers to use in teaching and reteaching grade-level content, and aligned 
benchmark or interim assessments to gauge what areas of the curriculum must be retaught prior 
to the end-of-year summative assessments intended for use as system accountability measures. 
Summative end-of-year assessments can no longer stand in as providers of the sole source of 
information for teachers and parents about student progress against grade-level content standards. 
Likewise, end-of-year assessments must no longer be the first time during a school year that a 
struggling student is assessed on the content covered in the state’s adopted standards. 

The contribution by Roach et al. (2009/this issue) provides useful guidance to states attempting 
to ensure curricular access for students with disabilities, by indicating the need to incorporate into 
state monitoring efforts indicators by which to track access to the general education curriculum, 
activities that will additionally help states document the consequential validity of the AA-MAS. 
The authors provide an intriguing review of the concept of “opportunity to learn” (OTL), its 
legislative history, and the role it has played in the history of school reform. Roach et al. also 
remind us that because the OTL provisions in the Goals 2000 Educate America Act were never 
instituted, OTL standards were made voluntary in the 1995 Elementary and Secondary Education 
Act. Thus, in the 30 years since the 1981 Debra P. v. Thurlington ruling, the need to provide 
opportunity to learn skills and concepts assessed by mandated achievement tests has been a 
continuing problem on a national scale. 

Under the provisions of NCLB, all students were ensured access to the same content standards 
(content standards developed for “all students”). Nevertheless, states proceeded to develop content 
standards that often resulted in barriers to the creation of aligned alternate assessments by 
specifying in the content standard an inherent standard of achievement. Some state content 
standards dictated a level of performance complexity and skill (such as a text difficulty level) that 
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effectively rendered test items correctly aligned to those standards inaccessible to some students 
with disabilities—including students who were not students with the most severe cognitive 
impairments. Because requirements for alignment prohibit the modification of content standards 
themselves, confounds between content and achievement standards pose a problem for students 
with disabilities that must be avoided during the development of Common Core State Content 
Standards to render such standards accessible to “all students.” 

Roach et al. (2009/this issue) remind us that previously proposed OTL standards had included 
the goal of aligning curricula, instructional practices, and assessments to a state’s content stan- 
dards. Without such instructional alignment, students cannot be expected to, and should not be 
able to attain proficiency on, an AA-MAS. The frameworks presented in their article for evaluat- 
ing alignment, as well as methods of rating instructional emphases in the enacted curriculum, will 
prove invaluable to educators in monitoring efforts as well as in the development of consequential 
validity studies pursuant to the implementation of the AA-MAS. 

Studies cited by Roach et al. support the beneficial effect of inclusive classrooms in maximizing 
outcomes in reading vocabulary, math computations, and math applications. They also suggest that 
two other factors—student engagement and academic learning time—be measured to assess the 
effectiveness of inclusion. Moreover, the authors propose inclusion of several domains of student 
engagement, rather than trying to treat engagement as a dichotomized construct. Likewise, the 
assessment of coverage of material during instruction is described. To transition students from 
prerequisite to on-grade-level skills, recent advances in delivery of aligned and differentiated 
instructional modules with built-in formative assessments hold promise as tools to foster both 
engagement in instruction and the extension of academic learning time for students eligible for 
the AA-MAS. 

Roach et al. (2009/this issue) write that historically the IEP development process has not 
consistently resulted in improved access to the general education curriculum for students with 
disabilities. Goals and objectives most often listed in students’ IEPs have not matched the 
curriculum and instruction provided in inclusive environments and do not adequately address 
objectives for reading comprehension or number concepts. Future studies are needed to monitor 
the effects of the presence of “standards-based” IEP goals on inclusion, curricular access, and 
academic engagement. Roach et al. have resurrected the heuristic role that studying opportunity to 
learn and curriculum access can play in the evaluation of the impact of educational programming 
changes of all kinds. 

Although of late, some disability advocates have been cautious about the unintended conse- 
quences of even identifying a population of students eligible for an AA-MAS, contending that 
there are also persistently low-achieving students without disabilities and plenty of room for 
improvement in providing opportunities to learn for such students. In response, it is important 
to address factors other than disability, including mobility, lack of motivation, and engagement 
as well as lack of opportunity to learn, which may play a role in persistent low achievement. 
The interventions and supports appropriate to these different scenarios, including the assessment 
options required by different groups of students, are, however, distinguishable. The impact of a 
student’s disability alone must be the determining factor in eligibility decision making for the 
AA-MAS. 

It is only by assuring that all “tier one” instruction is fully aligned to state-adopted content 
standards for the grade and that “tier two” evidence-based interventions are implemented in a 
Response to Intervention format that states can ensure that the learning needs of all struggling 
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students in general education programs are met. At the same time, technical development of the 
AA-MAS must reflect the specific learning needs of the target group of students with disabilities. 


DEVELOPING A MORE ACCESSIBLE ASSESSMENT 


Egan et al. (2009/this issue) provide a comprehensive review of considerations that should guide 
the formulation of performance level descriptors and other standard setting methods applied to 
the AA-MAS. As the authors indicate, a fundamental concern in standard setting is the lack 
of familiarity among test designers and psychometricians with this target examinee population. 
They identify the information that can be gleaned from understanding both the enacted and 
the intended curriculum appropriate to eligible students. The AA-MAS should be designed to 
provide teachers and parents with an accurate picture of their progress against the same grade- 
level content standards measured on a general assessment, and the definition of a modified 
performance standard must be developed in the context of the entire framework of performance 
level descriptors within a balanced and integrated assessment system (e.g., Gong, 2009). 

Egan et al. also indicate an additional potential benefit of AA-MAS, which is its potential within 
a comprehensive assessment system to provide a missing link whereby students are enabled to 
“sraduate” from the AA-AAS to the AA-MAS, or from the AA-MAS to the regular assessment. 
As many teachers of students eligible to take an AA-AAS have reported, students with disabilities 
often surpass expectations when provided an opportunity to learn academic content aligned to 
appropriate grade-level content standards. No student should be presumed incapable of graduating 
from a lower to a higher achievement standard. 

AA-MAS must satisfy two essential technical requirements—it should improve access to the 
assessment for the target group, over and above the access to the regular assessment such students 
could attain with appropriate accommodations, and it should permit a more accurate assessment of 
the grade-level knowledge and skills such students have acquired. The modifications represented 
in the test must appropriately reduce the complexity of the cognitive processing demanded by the 
test and must establish more reasonable expectations for content mastery. 

As Egan et al. (2009/this issue) advise, the development of modified academic achievement 
standards should illuminate the content that should be taught to eligible students while reflecting 
the “depth” to which the associated content can be learned within the course of a school year. In 
addition, modified achievement standards must also pertain to the mastery of content standards 
intended for “all students” in the enrolled grade. These authors provide practical direction in 
articulating the relationship of performance levels to grade-level achievement from the inception 
of policy-based standards, through their development as empirically based performance level 
descriptors. 

Egan et al. also propose several practical methods to help states decide how to accomplish the 
establishment of performance standards on the AA-MAS and the linkage of those standards to 
the general assessment, with considerations to guide the vertical articulation of cut scores. The 
authors also provide useful descriptions of the limitations associated with each approach. 

Further adding to the considerable value of this issue of the Peabody Journal of Education, 
Kettler, Elliott, and Beddow (2009/this issue) introduce a pragmatic tool by which to measure 
the accessibility of test items which promises to be useful in developing or refining test items 
for a variety of target populations. The Test Accessibility and Modification Inventory (TAMI) 
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was developed to address the problem of understanding item accessibility, given that known 
psychometric properties associated with test items are generally derived from item trials on 
the general population. Although developed in response to the AA-MAS, the TAMI would 
appear to have potential utility in other assessment contexts, such as increasing accessibility 
of general assessments by permitting quantification and measurement of item-accessibility for 
subpopulations of students with disabilities or for English language learners. 

Kettler et al. (2009/this issue) describe a theoretically derived approach based on Cognitive 
Load theory, which has roots in the work of George Miller, Paas et al.’s triune model of cognitive 
load, Clark et al.’s (2006) guidelines for maximizing the efficiency of learning, Haladyna et 
al.’s guidelines for construction of items and response options, and Rodriguez et al.’s work on 
distractor analysis. The TAMI consists of both an item analysis and test analysis component, and 
assists in appraising the structural complexity of test items, separating sources of item complexity 
(germane, intrinsic, and extraneous), and permitting the evaluation of item alignment to grade- 
level content standards. The resulting tool promises to be helpful in designing accessible test 
items to enable students to make more efficient use of limited attentional and working memory 
capacity during the test-taking process. Item development, especially when intended to improve 
access for students with disabilities, will benefit from the contribution of cognitive psychologists, 
both to advance our understanding of the nature of cognitive processing challenges encountered 
on test items by students with disabilities as well as to minimize unnecessary barriers created by 
test items which demand cognitive processing extraneous to the content being measured. 

Although it is critical when examining item characteristics for modification that research 
be carried out on a sample of students with disabilities carefully screened for a history of 
opportunity to learn the underlying content being assessed, so as to avoid confounding the effects 
of accessibility and instruction, the TAMI and the general modification paradigm introduced 
by these authors will no doubt prove to be an important tool in the quest for more accessible 
tests. The authors’ approach to the study of student cognitive processing for purposes of item 
development, including use of student focus groups and follow-up teacher interviews presents a 
research paradigm which can improve understanding of item accessibility-requirements for an 
AA-MAS. 

Over the past 2 years, concerted efforts have been made by state departments of education 
across the country to address long-standing gaps in the availability of accessible assessments 
and opportunities to learn. Together with innovative programming to support standards-aligned 
instruction for all students with disabilities, the development and implementation of more ac- 
cessible and more challenging alternate assessments is vital to ensure continued and meaningful 
educational progress for students with disabilities. The articles in this special issue of the Peabody 
Journal of Education make valuable contributions to the furtherance of these important objectives. 
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Psychometric Considerations for Alternate Assessments 
Based on Modified Academic Achievement Standards 


Michael C. Rodriguez 
Educational Psychology, University of Minnesota 


Because of the unique nature of the students eligible for alternate assessments based on modified 
academic achievement standards, their varied access to the general education curriculum, and their 
unique learning needs, innovative psychometric thinking and practice is needed to assure high tech- 
nical quality of alternate assessments. Indeed, we at least must marshal state-of-the-art procedures to 
secure strong psychometric evidence to support appropriate and meaningful design and use of these 
important assessments. The authors contributing work to this special issue, Alternate Assessments 
Based on Modified Academic Achievement Standards, address important issues and provide guid- 
ance to policymakers, test developers, and educators. They also each raise important technical quality 
issues. This article offers a brief review of such psychometric considerations, in light of the work and 
comments of the special issue authors. 


This special issue of the Peabody Journal of Education on Alternate Assessments Based 
on Modified Academic Achievement Standards: New Policy, New Practices, and Persistent 
Challenges provides researchers, educators, and policymakers with a set of important consid- 
erations and guidance in our current exploration of the world of alternate assessment design and 
implementation. Although not explicitly intended, the contributors to the special issue present 
issues that have implications for the technical quality of alternate assessments based on modified 
academic achievement standards (AA-MAS). To focus attention, I briefly review the technical 
issues raised. I then turn to psychometric considerations to provide guidance for establishing a 
high level of quality evidence to support some of the more ambitious claims about AA-MAS. 


CONTRIBUTORS RAISE TECHNICAL QUALITY ISSUES 


Authors of all six articles in this special issue present important challenges to the technical 
quality of AA-MAS. Some of this attention is due to the language in the federal guidance and 
peer review documentation. Guidance on AA-MAS development allows for the use of modified 
versions of the general education assessment or the development of new tests and the use of 
easier achievement standards and easier test questions. At the same time, the content covered 
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on the alternate assessment must be the same as the géneral assessment (requiring alignment 
with grade-level content standards). In their introduction to this special issue, Kettler, Elliott and 
Beddow (2009/this issue) raise three challenges in these requirements, including the identification 
of qualified students, the degree of appropriate modification to achievement standards, and the 
demands of adequately documenting technical quality. ‘These challenges have psychometric 
implications and are reflected in the comments of the other authors. 

Identification is the focus of the work by Zigmond and Kloo (2009/this issue). After a review 
of legislation leading up to the inclusion of students with disabilities in statewide assessment 
programs and the authorization for alternate assessments, Zigmond and Kloo examine decisions 
that states encounter to identify the appropriate population for the AA-MAS. They raise impor- 
tant questions regarding the low proficiency rates of students on the general assessment as a 
reflection of ability versus access or the presence of construct-irrelevant variance. With respect 
to the fit of the assessment, construct-irrelevant variance is a serious source of interference with 
test interpretation, misinforming subsequent decisions. Herein is a challenge to states: create a 
meaningful alternate assessment to meet the assessment needs of students with disabilities that 
considers their unique instructional and learning needs. 

Lazarus and Thurlow (2009/this issue) identify changes in states’ AA-MAS between 2007 and 
2008, based on information made available at state Web sites and follow-up with state personnel. 
They found online information for six states in 2007 and nine in 2008 with AA-MAS in place 
or under development. Regarding the characteristics of the assessments in 2008, they found all 
nine states included multiple-choice items, two had constructed-response items, five employed a 
writing prompt, and one included performance tasks. Lazarus and Thurlow also review methods 
of test design of the nine states: eight used fewer items, six removed a multiple-choice distractor, 
six employed simplified language throughout, five had fewer reading passages, and five had 
shorter reading passages. In addition, five states used fewer items per page, five used larger font 
size, three employed segmentation of passages, and two bolded or underlined key text. States are 
constructing their AA-MAS within the framework of their existing assessment programs. 

A core concept in the development of AA-MAS and the intent of alternate assessments for 
students with disabilities overall is increasing access to grade-level general education curriculum. 
Roach et al. (2009/this issue) tackle this overriding goal. In all of the legislation and guidance 
documentation, the issue of access to the general curriculum is prominent, expected, and in many 
cases assumed. But as most of the authors acknowledge in regard to their specific topic, the 
students for whom AA-MAS would be most appropriate may not have full access to the general 
curriculum. Roach and colleagues aptly review the contexts, opportunities, and information needs 
to address curricular access for all students. They point to the need for a multidimensional measure 
of student engagement in general education programming, which can play an important part in 
interpreting the results of AA-MAS, improving our ability to make meaningful inferences about 
student achievement of grade-level content standards. 

The task of taking an existing assessment system through an appropriate modification pro- 
cess (while maintaining the coherence between instruction, learning, and assessment for the 
students who would benefit the most) is a daunting challenge. Kettler, Elliott, and Beddow 
(2009/this issue) have undertaken this challenge in the development of their theory-guided data- 
based approach to design and modification. The Test Accessibility and Modification Inventory 
(TAMI) presents systematic guidance to developing, modifying, and evaluating items and tests 
to maximize accessibility and improve measurement. It is based on principles of cognitive load 
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theory, universal design, and item writing research and guidelines—essentially better measure- 
ment. The TAMI could easily be integrated into the development process of all assessments. The 
cognitive lab and experimental studies based on the use of the TAMI provide strong support for its 
use. Kettler, Elliott, and Beddow also review important sources of validity evidence based on the 
framework in the Standards for Educational and Psychological Testing (American Educational 
Research Association, American Psychological Association, & National Council on Measure- 
ment in Education, 1999), including construct-related evidence, and evidence related to content, 
response processes, internal structure, relations to other variables, and the consequential bases of 
validity. They support their work with references to the Standards and empirical research based 
on the principles used to develop the TAMI, providing a strong basis for its use. 

Continued use of an assessment in an accountability system requires the adoption of perfor- 
mance standards, including the development of performance level descriptors. Egan, Ferrara, 
Schneider, and Barton (2009/this issue) address the requirements to accomplish full implemen- 
tation of AA-MAS. They argue that the technical quality of AA-MAS is critical, including 
evidence to support the validity of inferences about academic achievement of grade-level content 
standards, which support the decisions resulting from the application of performance standards. 
They also discuss the challenges in following standard procedures, as the unique characteristics 
of AA-MAS require innovative thinking. Some of the challenges—for example, that many state 
AA-MAS are shorter than general assessments—should be considered limitations of the methods 
we have developed for longer assessments; many methods, such as alignment and standard setting 
methodologies, rely on the presence of a great deal of information in long standardized tests. 
These limitations should not be construed as limitations of AA-MAS but limitations in existing 
methodologies to establish the technical quality of AA-MAS. Unfortunately, Egan and colleagues 
suggest that the process of gathering validity evidence “is conceptually complex and requires 
significant sustained effort” (p. 573). Certainly validation requires sustained effort and should 
be ongoing. The idea that it is conceptually complex is not helpful and points to the inability of 
validity theorists to communicate clearly to those who need strong guidance the most. 

States continue to tackle these and many other challenges to provide access to high quality 
content and instruction and assess student performance. Palmer (2009/this issue) surveys as- 
sessment personnel from 22 states regarding their experiences with AA-MAS implementation. 
She notes that state personnel are looking for consistent guidance on the technical specifications 
necessary for acceptable AA-MAS. States appear uncertain about federal requirements and the 
effectiveness of AA-MAS. A strong example of these concerns is embodied in one respondent’s 
comments: “We’d need to conduct research on comparability, validity, reliability, impact, use of 
accommodations, etc.—all of that costs quite a bit. We’d need to conduct a standard setting, and 
do an alignment study” (p. 581). States recognize the importance of technical quality and their 
limited resources to provide such evidence. 


A PSYCHOMETRIC REVIEW 


There are multiple psychometric considerations raised in this special issue, and each one deserves 
another manuscript. For example, Egan and colleagues make a psychometric suggestion to support 
standard setting with AA-MAS, by augmenting the ordered item booklet used in some standard 
setting procedures. If the use of fewer items (typical in AA-MAS) results in limited coverage of the 
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ability continuum, augmenting the item booklet could improve the standard setting task—because 
the task of the standard setting panel is not to identify the number of items necessary to reach a 
certain performance level, but the ability that is associated with that level (based on performance- 
level descriptors). So if augmenting the item booklet improves the ability continuum coverage, 
particularly in the regions associated with the thresholds between the performance levels, the 
resulting cut scores are better identified and supported. 

This is related to a point made by Kettler, Elliott, and Beddow regarding validity evidence 
based on internal structure. They suggest that item response theory (IRT) is useful to improve 
the match between ability assessment and item difficulty. IRT provides a model to estimate the 
amount of information provided by test items across the ability continuum (the item characteristic 
curve). This can be used to construct a test with specified information loads across the ability 
scale by selecting items to achieve those specifications. 

To review all of the psychometric considerations made throughout the issue would be beyond 
the charge of this review. Instead, I see two areas as being most helpful as we move forward in 
the design and use of strong AA-MAS. First, there is a fair amount of discussion in the field of 
educational measurement regarding validity frameworks, what those frameworks should include, 
and how they should be used. Second, perhaps there is not enough attention paid to test score 
quality, as it is the test score on which inferences are based. 


CONSIDERATIONS FOR A PSYCHOMETRIC FRAMEWORK FOR 
AA-MAS 


In the work on AA-MAS in this issue and elsewhere, psychometrics have focused on the impor- 
tance of validity evidence. Validity evidence in an assessment system is perhaps local, at the point 
at which inferences and uses of assessment results are meaningful and appropriate. As such, a 
framework for AA-MAS validation and technical quality assurance must be local and focus on 
the immediate and future needs of the assessment program for students, families, schools, and 
states. 

Consistent with the measurement field’s current thinking about validity and technical quality 
more generally, every aspect of an assessment, from conceptualization and design to administra- 
tion, reporting, and use, requires quality evidence. Federal guidance requires AA-MAS to meet 
the same technical quality standards as general assessments, as they should. Alternate assess- 
ments are important tools for accountability, program evaluation, and decision making regarding 
student achievement. Because of the unique nature of alternate assessments, the use of the same 
psychometric criteria may not be appropriate; however, there is no reason why alternate assess- 
ments need to result in lower technical quality. It is important to support the use of alternate 
assessments with the highest level of technical quality and psychometric standards, perhaps even 
beyond that achieved by general assessments, particularly for eligible students. The goal is to 
provide eligible students with test scores that are of higher quality than they would be from the 
general assessment. This may require small-scale experimental designs, sampling eligible and 
noneligible students to take both versions of the assessment. 

Because of the high-stakes nature of alternate assessments, their unique characteristics, their 
relative infancy in the field of educational assessment, and the unique learning needs of eligible 
students, much of the effort to establish technical quality has focused on content and consequences. 
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Content is clearly an important characteristic, because for most achievement assessments, the 
primary inference is about content knowledge and skills—content-related validity evidence is 
a core concern. And to some extent, the consequential-basis of validity has been a focus in 
evidence gathering activities—perhaps because of the unique learning needs and curriculum of 
eligible students. Although these are important inferences and provide appropriate evidence for 
the interpretation and use of test results, the technical quality of the test scores themselves must 
not be overlooked. 

Psychometric evidence regarding the quality of scores can be gathered in many ways, including 
distractor quality (if multiple-choice), item quality, and test score quality. Distractor quality is 
typically identified through two metrics, proportion selecting the distractor and the distractor-total 
point-biserial correlation (a discrimination index for distractors). To be an effective distractor, 
each distractor should be selected approximately at the same rate (as each distractor should 
be plausible and potentially contain useful information regarding misconceptions or common 
errors), but probably at least by 5% of the respondents (given the overall difficulty of the item). 
Also, selection of a distractor should be negatively correlated with the total score (associated 
with lower total scores), where the point-biserial correlation should be less than —.20. In addition, 
IRT can provide even stronger evidence of distractor functioning, through a differential item 
functioning model for distractors, addressing differential distractor functioning across important 
subgroups (Green, Crone, & Folk, 1989; Middleton & Cahalan Laitusis, 2007). Qualitative 
evidence is equally useful when examining distractors, including evidence of cognitive aspects of 
distractors through think-aloud methodologies: Are the distractors eliciting the kinds of cognition 
we anticipated and value? 

Psychometric quality evidence at the item level includes information about item difficulty, 
item discrimination, and in an IRT model, item location and item fit. Pragmatically, we need 
items to cover a range of difficulty, including very easy items to motivate students and more 
difficult items to challenge students. But there are two concerns regarding this common guidance. 
First, classical item difficult (proportion correct) is relative, depending on the ability level of the 
responding students. In classical analyses, maximum efficiency is obtained when items are at 
about mid-difficulty, somewhere between chance level and 1.0. The current guidance that items 
may be “easier” is also relative, suggesting that for eligible students (and by implication, all 
students), items on the AA-MAS may be easier than items on the regular assessment, which 
will result in higher scores. Second, it is more important to be true to the intended test content 
and depth of knowledge (or cognitive task of items), because the most useful inferences from 
achievement tests are about subject matter knowledge and skill. 

Item discrimination (point-biserial correlations) should be positive and greater than .30. Dis- 
crimination is naturally limited for the most difficult and the easiest items—as it is a correlation 
metric and depends on variability. Similarly, when students are guessing, there is no relation 
between item score and total score (discrimination goes to 0). 

However, IRT presents a more practical model where item location (IRT difficulty) can be 
matched to the location of persons on the ability scale. Another common strategy for the location 
of items attempts to select enough items at the cut-point location so that greater precision is 
obtained at the critical points on the ability scale (the decision points). These are the principles 
that make computer adaptive testing so effective. IRT models provide a test of item fit to the 
model—that is, whether responses to an item fit the IRT model, such that individuals with lower 
ability tend to get the item incorrect more so than individuals with higher ability. As ability 
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increases, the probability of correctly responding increases; if not, item responses are likely 
influenced by construct-irrelevant factors, and the item fit indices indicate misfit. Again, methods 
such as think-alouds can contribute heavily here. 

Downing and Haladyna (1997) provided guidance for gathering item validity evidence, not- 
ing the importance of supporting score meaning at the item level. Among their advice, they 
suggested gathering evidence about (a) item writer training (documentation of training mate- 
rials and methods), (b) item-writing principles used (evidence of compliance with rules and 
item review processes), (c) cognitive behavior and depth of knowledge (the cognitive clas- 
sification system and documentation of its use), (d) item editing procedures (including the 
credentials of editors and results of item review), (e) item tryout results (including item field 
tests, cognitive lab, and experimental test results), and (f) key validation (documentation of the 
process to verify the key). The TAMI can accomplish much of this (Kettler et al., 2009/this 
issue). 

Finally, test score quality is important to establish because it is the test score that is used as 
the basis for inferences about levels of achievement. Test score quality is often characterized in 
terms of precision and accuracy. Precision is a matter of score consistency or score reliability 
and accuracy is a matter of score validity or hitting the target of measurement (what we intended 
to measure). Regarding precision, we typically employ indices of reliability and the standard 
error of measurement, from which we create score intervals acknowledging the imprecision of 
measurement. 


Rethinking Reliability 


The most commonly reported reliability coefficient is alpha. However, coefficient alpha assumes 
essentially tau-equivalent measurement, an assumption regarding the nature of item true scores 
and error scores that is rarely tested. When item standard deviations vary a great deal (suggesting 
that some items are measured on a different scale), when there are a relatively small (or smaller 
in comparison) number of items, and when there are multiple response formats, we likely violate 
the assumption of tau-equivalence, where alpha underestimates reliability. A more accurate 
estimate can be obtained through the use of a more appropriate measurement model, which 
matches the measurement properties of the test (e.g., a congeneric model that assumes each item 
measures the same construct, but possibly with different scales and different levels of precision). 
Graham (2006) reviewed procedures to obtain model-based estimates of reliability in a structural 
equation model approach that estimates the most appropriate measurement model for a given 
test. 

Cronbach (2004) and others have argued that alpha is not a particularly useful or appropriate 
estimate of reliability, given its hypersensitivity to group variability and numbers of items (asso- 
ciated with the assumption of tau-equivalence). A more comprehensive and sensitive estimate of 
measurement error is obtained through Generalizability Theory. Generalizability Theory allows 
us to partition measurement error to multiple sources of variation introduced because of the nature 
of the measurement procedure, which results in the most flexible means for estimating a wide 
range of reliability coefficients (including alpha). 
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Rethinking Construct Validity as a Unifying Framework 


It is important to target the collection of validity evidence to defend the most important claims we 
hope to make from the results of an AA-MAS to defend the accuracy of measurement. However, it 
is not clear that a focus on construct validity (in its unitary conceptualization) is helpful. Messick 
(1989), following the lead of Cronbach and Meehl (1955) and others, argued for a unified model 
of validity, under the umbrella of construct validation. Many measurement specialists have since 
argued against the daunting demands of construct validation as being beyond the capacity of 
the typical test developer as well as test takers and test users. Even Cronbach (1989), 34 years 
following his seminal article on construct validity, apologized for advancing such a limiting 
view of test score meaning: “It was pretentious to dress up our immature science in positivist 
language; and it was self-defeating to say that a construct not part of a nomological network 
is not scientifically admissible” (p. 159). Some have argued for the primacy of content-related 
evidence for achievement tests, because the target of measurement is subject matter knowledge and 
skill (e.g., Lissitz & Samuelsen, 2007). Others point to the important contributions of cognitive 
psychologists and suggest a stronger focus on score meaning (e.g., Gorin, 2007). Some argue 
for a more pragmatic approach, one that addresses the claims and assumptions made in score 
interpretation and use (e.g., Kane, 2002). 

In a comprehensive modern perspective on validity, all evidence that can be gathered to bear 
on test score interpretation is useful, including reliability evidence, content evidence, response 
processes, the internal structure of the test, and relations to other variables (elements mostly 
included in the Standards). There remains significant debate regarding the role of consequences 
and the degree to which they bear on the accuracy of test scores. Scriven (2002) for one, argued 
that issues related to test score use are important but are more about the utility of test scores 
rather than the meaning of test scores. Nevertheless, evidence that supports inferences and uses 
of test scores (to the extent that uses are supported by the meaningfulness and appropriateness 
of a particular test score) is important to document and report, particularly in the context of 
high-stakes testing, regardless of whether it is gathered under the auspices of validity. 

For the purposes of AA-MAS, there are federal guidelines regarding the basic validity-related 
evidence that must be gathered; the authors in this volume have contributed to our understanding 
of these guidelines. But for schools, educators, families, and students, perhaps there are more 
basic requests regarding the information value from AA-MAS, including a clearer understanding 
of the meaning of scores and descriptions of the cognitive performance producing those scores. 
Although the current demands on AA-MAS are federal, the real importance of these assessments 
in the lives of students is local. 
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